Why RGB Is a Bad Choice for Robotics Perception

If you've ever built a robot that uses a camera to detect objects, you've probably seen this problem. You test everything on your desk, tune the thresholds, and the robot reliably detects a red object. The bounding box is stable, tracking works, and the demo looks perfect. Then the environment changes slightly. Someone opens a window, a cloud passes over the sun, or a shadow falls across the object. The object hasn't moved and its physical appearance hasn't changed, but the detector starts flickering. The bounding box becomes smaller, moves around, or disappears completely.

Nothing crashed. OpenCV didn't throw an exception. The robot simply stopped recognizing the object.

The obvious reaction is to change the threshold values, add more filtering, or tune the system until it works again. But that usually treats the symptom rather than the actual problem. The deeper issue is that the perception system is working with a representation of the image that is highly sensitive to illumination.

A Camera Doesn't See a Red Ball

This is one of the most important things to understand when working with computer vision: a camera doesn't see objects. It measures light.

A camera converts the light reaching its sensor into numerical values. For example, an OpenCV image with a resolution of 1920 × 1080 is essentially a three-dimensional array containing more than two million pixels, with three channel values for each pixel.

image.shape

(1080, 1920, 3)

OpenCV normally stores those channels as BGR rather than RGB:

[B, G, R]

So when your robot receives an image, it doesn't receive something that says "there is a red ball here." It receives millions of numerical measurements. Everything your perception pipeline does afterward—segmentation, contour detection, tracking, object localization—is an attempt to transform those measurements into something meaningful.

That distinction becomes important when the lighting changes.

Follow One Pixel

Imagine a pixel on a red plastic ball. Under bright indoor lighting, the camera might produce something like:

[30, 30, 240]

The blue and green channels are relatively low, while the red channel is high. A simple RGB-based detector can easily classify that pixel as red.

Now put the same ball under a shadow. The ball itself hasn't changed, but the amount and characteristics of light reaching the camera have changed. The camera might now measure:

[20, 20, 120]

The physical object is still red, but the numerical measurement is very different.

This is the fundamental weakness of using raw RGB values for color segmentation. RGB does not directly represent an object's intrinsic color. It represents the response of the camera's sensors to incoming light. Change the illumination and the measured values can change significantly.

That's why a threshold that worked perfectly five minutes ago can suddenly fail.

Why RGB Thresholding Breaks

Consider a simple detector:

mask = cv2.inRange(image, lower_red, upper_red)

Suppose you tuned it under bright lighting and effectively decided that pixels with a red-channel value above 200 should be considered red.

Under the original lighting:

R = 240

The pixel passes the threshold.

Now a shadow falls across the object:

R = 120

The exact same physical object now fails the threshold.

The computer doesn't know that the lighting changed. It doesn't reason about the object remaining physically red. It simply evaluates the numerical condition:

120 < 200

So the pixel is rejected.

If enough pixels are rejected, the binary segmentation mask becomes smaller. The contour extracted from that mask changes shape, the bounding box changes, and anything downstream—such as object tracking or robot control—can become unstable.

This is why perception bugs often propagate through an entire robotics system. A small error at the sensor representation level can eventually become a large error in the robot's behavior.

The Real Problem Is Illumination

RGB mixes several things together that we often want to reason about separately: object color, illumination, exposure, reflections, camera gain, and white balance.

If the lighting changes, the RGB values can change even when the object itself doesn't.

This makes absolute RGB thresholds fragile. They work well when the environment is controlled, but robotics rarely operates in perfectly controlled environments.

A robot running in a laboratory with fixed lighting is one thing. A robot operating in a warehouse, outdoors, or even a different room has to deal with much more variation.

This is where choosing the right representation becomes important.

Why HSV Can Help

One common approach is to convert the image from BGR into HSV:

hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

HSV represents the image using three components: Hue, Saturation, and Value.

Hue describes the dominant color. Saturation describes how strongly that color is expressed, while Value represents brightness.

The useful property here is that color information is represented separately from brightness-related information. If a red object becomes darker because a shadow passes over it, the Value component can change substantially while the Hue can remain relatively similar.

That gives us a representation that is often more useful for color-based segmentation.

For example, instead of trying to detect a red object using absolute RGB values, you can define a region in HSV space:

mask = cv2.inRange(
    hsv,
    lower_red,
    upper_red
)

The detector is now primarily looking for pixels whose hue falls within the expected red range, while also constraining saturation and brightness so that extremely dark or nearly gray pixels aren't treated as valid red pixels.

This doesn't make the detector immune to lighting. It simply makes the representation better aligned with the problem you're trying to solve.

HSV Is Not a Magic Fix

It's important not to turn this into another oversimplification: HSV is not inherently "lighting invariant."

Strong shadows, reflections, extremely low light, camera auto-exposure, and low saturation can still cause problems. When saturation approaches zero, for example, Hue becomes poorly defined because the pixel is effectively gray.

So the engineering lesson isn't:

RGB is bad, always use HSV.

The better lesson is:

Choose a representation that separates the information you care about from the variations you don't care about.

Sometimes that means HSV. In other situations, another color space, normalization technique, adaptive thresholding, feature-based detection, or a learned vision model may be more appropriate.

The representation should follow the problem.

From Pixels to Robot Decisions

A simple color-based perception system might look like this:

Camera
  ↓
Image
  ↓
Color-space conversion
  ↓
Segmentation
  ↓
Binary mask
  ↓
Morphological filtering
  ↓
Contours / connected components
  ↓
Object position and size
  ↓
Tracking
  ↓
Robot control

Each stage transforms the data into a representation that is more useful for the next stage.

The camera gives you raw measurements. Segmentation turns those measurements into candidate regions. Geometry gives you information such as position, area, and shape. Tracking adds temporal information. Finally, the control system uses that estimated state to decide how the robot should move.

If the first representation is fragile, everything downstream inherits that fragility.

That's why perception engineering isn't just about finding the right OpenCV function. It's about understanding what your sensor measures and deciding which parts of that measurement are actually useful for the task.

The Bigger Robotics Lesson

This principle goes beyond color detection.

When a robot suddenly stops working, the first instinct is often to look for a software bug. But robotics systems operate on physical measurements, and physical measurements are noisy and variable. Lighting changes. Sensors drift. Camera exposure changes. Objects become partially occluded. Motors wear. Mechanical vibration introduces noise. Communication adds latency.

So a robust perception system cannot depend on ideal sensor inputs.

When something breaks, a better question is not simply "what changed in the code?" It is "what changed in the data, and what assumption did my algorithm make about that data?"

That mindset is what separates a working computer-vision demo from a perception system that can survive outside the developer's desk.

The robot was never confused by the shadow.

The shadow simply exposed an assumption in the perception pipeline.