Can You See Me Now? Adding Computer Vision to Your Arduino

Intermediate Course · Week 4

Sketchnotes, week 4

Lesson Goal	How do you add a camera to your Arduino Uno Q and build a pipeline that sees, classifies, and talks about what it sees using local AI?
What you'll learn	By the end of this week you will be able to: - Connect and initialize the mini camera with the Uno Q - Capture a still image and transfer it over WiFi to your laptop - Run MobileNet (from Week 2) on the laptop to classify the image - Chain vision + language models: classification → LLM prompt → useful response - Display multi-modal AI results on your laptop screen with LED feedback on the Uno Q
Tools you'll need	Arduino Uno Q board, USB cable, mini camera, mini mouse, USB hub, laptop with Arduino App Lab installed, Python 3 with Flask + Pillow + TensorFlow, Ollama running (from Week 2), the Python server from Week 3
End result	A working "object explorer" — point the camera at any object, click the mouse, and the system identifies it, shows the label on screen, lights up the LED array, and displays an interesting fact generated by the local LLM
Time needed to complete	90 minutes

Session Plan

Part 1 — Camera Setup: Your Arduino Learns to See (20 min)

Mini-lecture: How a Camera Talks to a Microcontroller (5 min)

A camera like the Arducam Mini 2MP Plus communicates with a microcontroller over SPI (Serial Peripheral Interface) — a fast, synchronous data bus. Unlike USB cameras that do their own processing, this camera sends raw pixel data to the Uno Q, which then needs to send it somewhere else (like your laptop) to make sense of it.

The Arducam uses the OV2640 sensor, which can capture:

Still images up to 1600×1200 (2 megapixels)
JPEG-compressed images (smaller file size, faster to transfer)
Raw RGB data (larger, but no compression artifacts)

For our pipeline, we'll capture JPEG images — they're small enough to send over WiFi quickly.

Recall from Week 2: MobileNet was designed to be small and fast, running on devices with limited computing power. The Arducam + Uno Q is exactly the kind of setup MobileNet was made for — capture on a small device, classify on a nearby computer.

Activity: Connect the Arducam and Capture a Test Image (15 min)

Connect the camera to your Uno Q using the ribbon cable. The Arducam connects to the SPI pins on the Uno Q. Refer to the Arducam wiring guide for your specific board.
Install the Arducam library in Arduino App Lab:
- In App Lab, go to the Library Manager
- Search for "Arducam" and install the official Arducam library
Write a sketch to capture a test image:

cpp

#include "ArduCAM.h"
#include "SPI.h"
#include "Wire.h"

// Select the correct model for your camera
#define ARDUCAM_MODEL ArduCAM_OV2640

ArduCAM myCAM(ARDUCAM_MODEL, SS);

void setup() {
  Serial.begin(9600);
  while (!Serial) { ; }
  
  // Initialize I2C
  Wire.begin();
  
  // Initialize SPI
  SPI.begin();
  
  // Check if the camera is connected
  uint8_t vid, pid;
  myCAM.wrSensorReg8_8(0xff, 0x01);
  myCAM.rdSensorReg8_8(OV2640_CHIPID_HIGH, &vid);
  myCAM.rdSensorReg8_8(OV2640_CHIPID_LOW, &pid);
  
  if ((vid != 0x26) && ((pid != 0x41) || (pid != 0x42))) {
    Serial.println("Camera not detected!");
    while (1) { ; }
  }
  Serial.println("Camera detected!");
  
  // Initialize the camera
  myCAM.set_format(JPEG);
  myCAM.InitCAM();
  myCAM.set_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
  myCAM.clear_fifo_flag();
}

void loop() {
  // Capture an image
  Serial.println("Capturing image...");
  myCAM.flush_fifo();
  myCAM.clear_fifo_flag();
  myCAM.start_capture();
  
  while (!myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK)) {
    // Wait for capture to complete
  }
  Serial.println("Capture complete!");
  
  // Read the image size
  uint32_t size = myCAM.read_fifo_length();
  Serial.print("Image size: ");
  Serial.print(size);
  Serial.println(" bytes");
  
  delay(5000);  // Wait 5 seconds before next capture
}

Upload this sketch and open the Serial Monitor. You should see "Camera detected!" followed by the image size in bytes.

Troubleshooting: If the camera isn't detected, check your wiring. The Arducam ribbon cable can be fragile — make sure it's fully inserted on both ends. Try reseating the connection.

Discussion prompt: "What do you notice about the image size? How does it compare to a photo from your phone? Why might a smaller image be better for our AI pipeline?"

Part 2 — Image Over WiFi: Sending Pictures to Your Laptop (20 min)

Activity: Extend the Python Server to Accept Images (10 min)

Last week, your Python server accepted text prompts. Now we need it to accept images too. Update your arduino_bridge.py to add an image endpoint:

python

from flask import Flask, request, jsonify
import subprocess
import json
import tkinter as tk
from threading import Thread
import base64
import os
from datetime import datetime

app = Flask(__name__)

latest_response = "Waiting for a click..."
latest_image = None

@app.route('/')
def home():
    return "Arduino Bridge Server is running!"

@app.route('/ask', methods=['POST'])
def ask_llm():
    global latest_response
    data = request.json
    prompt = data.get('prompt', 'Say hello')
    
    result = subprocess.run(
        ['ollama', 'run', 'llama3.2', prompt],
        capture_output=True,
        text=True,
        timeout=30
    )
    
    latest_response = result.stdout.strip()
    return jsonify({'response': latest_response})

@app.route('/upload', methods=['POST'])
def upload_image():
    """Receive an image from the Uno Q and save it."""
    global latest_image
    
    data = request.json
    image_data = data.get('image', '')
    
    if image_data:
        # Decode base64 image
        image_bytes = base64.b64decode(image_data)
        
        # Save with timestamp
        filename = f"capture_{datetime.now().strftime('%Y%m%d_%H%M%S')}.jpg"
        with open(filename, 'wb') as f:
            f.write(image_bytes)
        
        latest_image = filename
        print(f"Image saved: {filename} ({len(image_bytes)} bytes)")
        return jsonify({'status': 'ok', 'filename': filename})
    
    return jsonify({'status': 'error', 'message': 'No image data'})

def create_window():
    """Display window for LLM responses."""
    root = tk.Tk()
    root.title("AI Desk Companion")
    root.geometry("600x400")
    
    label = tk.Label(root, text="Point the camera and click the mouse!", 
                     font=("Arial", 16), wraplength=550)
    label.pack(pady=20)
    
    response_text = tk.Text(root, font=("Arial", 12), wrap="word",
                            padx=10, pady=10)
    response_text.pack(fill="both", expand=True, padx=20, pady=10)
    
    def update():
        if latest_response != "Waiting for a click...":
            response_text.delete(1.0, tk.END)
            response_text.insert(1.0, latest_response)
        root.after(1000, update)
    
    update()
    root.mainloop()

Thread(target=create_window, daemon=True).start()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=False)

Install the required library:

bash

pip install pillow

Activity: Send an Image from the Uno Q to the Laptop (10 min)

Now write a sketch that captures an image and sends it over WiFi to your Python server:

cpp

#include "ArduCAM.h"
#include "SPI.h"
#include "Wire.h"
#include "WiFiS3.h"

#define ARDUCAM_MODEL ArduCAM_OV2640

ArduCAM myCAM(ARDUCAM_MODEL, SS);
WiFiClient client;

char ssid[] = "YourWiFiName";
char pass[] = "YourWiFiPassword";
char serverIP[] = "192.168.1.XXX";  // Your laptop's IP
int serverPort = 5000;

// Buffer for reading image data
#define BUFFER_SIZE 1024
uint8_t buffer[BUFFER_SIZE];

void setup() {
  Serial.begin(9600);
  while (!Serial) { ; }
  
  Wire.begin();
  SPI.begin();
  
  // Initialize camera
  myCAM.set_format(JPEG);
  myCAM.InitCAM();
  myCAM.set_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
  myCAM.clear_fifo_flag();
  
  // Connect to WiFi
  Serial.print("Connecting to WiFi");
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void loop() {
  // Capture image
  Serial.println("Capturing...");
  myCAM.flush_fifo();
  myCAM.clear_fifo_flag();
  myCAM.start_capture();
  
  while (!myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK)) { ; }
  
  uint32_t size = myCAM.read_fifo_length();
  Serial.print("Image size: ");
  Serial.println(size);
  
  // Send image to laptop over WiFi
  if (client.connect(serverIP, serverPort)) {
    // Read image data and encode as base64
    // (In practice, you'd read the FIFO and encode it)
    // For this example, we'll send a placeholder
    String json = "{\"image\":\"placeholder\"}";
    
    client.println("POST /upload HTTP/1.1");
    client.println("Host: " + String(serverIP));
    client.println("Content-Type: application/json");
    client.print("Content-Length: ");
    client.println(json.length());
    client.println();
    client.println(json);
    
    Serial.println("Image sent!");
    client.stop();
  } else {
    Serial.println("Connection failed");
  }
  
  delay(10000);  // Wait 10 seconds
}

Note: The full base64 encoding of a JPEG image on the Uno Q requires reading the camera's FIFO buffer byte by byte and encoding it. The Arducam library includes examples for this. For this lesson, the key concept is the pipeline: capture → encode → send over WiFi → receive on laptop.

Upload this sketch. Watch the Serial Monitor — you should see the image capture size and a confirmation that it was sent. Check your laptop — the Python server should print "Image saved: capture_....jpg"

Full-group debrief (5 min)

Facilitator-led discussion:

What was the image size? How long did it take to send?
What happens if you move the camera while it's capturing?
Why do you think we send the image as base64 over HTTP instead of raw bytes?

Part 3 — Vision Pipeline: Classifying What the Camera Sees (20 min)

Mini-lecture: How MobileNet Classifies Images (5 min)

In Week 2, you used MobileNet in the My Room app to classify objects in your browser. MobileNet is a convolutional neural network (CNN) trained on ImageNet — the dataset created by Fei-Fei Li, who we learned about in Week 1.

MobileNet can recognize 1,000 different categories of objects, from "water bottle" to "computer mouse" to "pizza." It was designed to be small enough to run on phones and embedded devices.

Today, we'll run MobileNet on your laptop (not on the Uno Q — the Uno Q captures the image, but the laptop does the heavy computation). This is the same client-server architecture we built in Week 3, but now the client sends images instead of text.

Activity: Add Image Classification to the Python Server (10 min)

Update your Python server to classify images using MobileNet:

python

from flask import Flask, request, jsonify
import subprocess
import json
import tkinter as tk
from threading import Thread
import base64
import os
from datetime import datetime
from PIL import Image
import numpy as np
import tensorflow as tf
from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2, preprocess_input, decode_predictions

app = Flask(__name__)

latest_response = "Waiting for a click..."
latest_image = None
latest_label = ""

# Load MobileNet model (this takes a moment)
print("Loading MobileNet model...")
model = MobileNetV2(weights='imagenet')
print("Model loaded!")

@app.route('/')
def home():
    return "Arduino Bridge Server is running!"

@app.route('/ask', methods=['POST'])
def ask_llm():
    global latest_response
    data = request.json
    prompt = data.get('prompt', 'Say hello')
    
    result = subprocess.run(
        ['ollama', 'run', 'llama3.2', prompt],
        capture_output=True,
        text=True,
        timeout=30
    )
    
    latest_response = result.stdout.strip()
    return jsonify({'response': latest_response})

@app.route('/upload', methods=['POST'])
def upload_image():
    """Receive an image, classify it, and return the label."""
    global latest_image, latest_label, latest_response
    
    data = request.json
    image_data = data.get('image', '')
    
    if image_data and image_data != "placeholder":
        image_bytes = base64.b64decode(image_data)
        
        # Save the image
        filename = f"capture_{datetime.now().strftime('%Y%m%d_%H%M%S')}.jpg"
        with open(filename, 'wb') as f:
            f.write(image_bytes)
        
        latest_image = filename
        
        # Classify the image with MobileNet
        img = Image.open(filename)
        img = img.resize((224, 224))  # MobileNet expects 224x224
        img_array = np.array(img)
        img_array = np.expand_dims(img_array, axis=0)
        img_array = preprocess_input(img_array)
        
        predictions = model.predict(img_array, verbose=0)
        decoded = decode_predictions(predictions, top=1)[0][0]
        
        label = decoded[1]  # Class name (e.g., "water_bottle")
        confidence = decoded[2]  # Confidence score (e.g., 0.95)
        
        latest_label = f"{label} ({confidence:.1%} confidence)"
        
        # Generate an LLM response about the object
        llm_prompt = f"I see a {label}. Tell me one interesting fact about this object."
        result = subprocess.run(
            ['ollama', 'run', 'llama3.2', llm_prompt],
            capture_output=True,
            text=True,
            timeout=30
        )
        
        latest_response = f"**I see: {label}**\n\n{result.stdout.strip()}"
        
        return jsonify({
            'status': 'ok',
            'label': label,
            'confidence': float(confidence),
            'response': latest_response
        })
    
    return jsonify({'status': 'error', 'message': 'No image data'})

def create_window():
    """Display window with classification and LLM response."""
    root = tk.Tk()
    root.title("AI Desk Companion")
    root.geometry("600x500")
    
    label = tk.Label(root, text="Point the camera and click the mouse!", 
                     font=("Arial", 16), wraplength=550)
    label.pack(pady=20)
    
    response_text = tk.Text(root, font=("Arial", 12), wrap="word",
                            padx=10, pady=10)
    response_text.pack(fill="both", expand=True, padx=20, pady=10)
    
    def update():
        if latest_response != "Waiting for a click...":
            response_text.delete(1.0, tk.END)
            response_text.insert(1.0, latest_response)
        root.after(1000, update)
    
    update()
    root.mainloop()

Thread(target=create_window, daemon=True).start()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=False)

Install TensorFlow if you haven't already:

bash

pip install tensorflow

Note: TensorFlow is a large package. If the download is slow, you can use a smaller alternative like tflite-runtime or run the classification with a separate Python script that the server calls.

Activity: Test the Full Vision Pipeline (5 min)

Make sure your Python server is running with the updated code
Make sure your Uno Q is connected to WiFi and the camera is connected
Point the camera at an object (a water bottle, a book, a plant — anything)
Watch the Serial Monitor on the Uno Q for the capture confirmation
Check your laptop — the server window should show the classification label and an interesting fact from the LLM

Try these objects:

A computer mouse (the model should recognize it)
A phone
A cup or water bottle
A plant or flower
Your own face (the model can detect "person")

Discussion prompt: "Were there any objects the model misidentified? Why do you think that happened? What does this tell you about the training data?"

Mini-lecture: What Is Multi-Modal AI? (5 min)

So far, you've used:

Text-only AI (Week 3): Send a text prompt → get a text response
Vision-only AI (Part 3 above): Send an image → get a classification label

Now you're going to combine them into a multi-modal pipeline — AI that processes multiple types of input (vision + language) and produces multiple types of output (text + LED patterns).

This is how real-world AI systems work:

A self-driving car sees the road (vision) and decides to turn (action)
A medical AI looks at an X-ray (vision) and generates a report (language)
A smart home camera sees a person (vision) and announces who it is (language + sound)

Your Uno Q is now a multi-modal AI device!

Activity: Add LED Feedback Based on Classification (10 min)

Update your Uno Q sketch to light up the LED array based on what the camera sees. The Python server will send back a color code along with the classification:

cpp

#include "ArduCAM.h"
#include "SPI.h"
#include "Wire.h"
#include "WiFiS3.h"
#include "Arduino_LED_Matrix.h"

#define ARDUCAM_MODEL ArduCAM_OV2640

ArduCAM myCAM(ARDUCAM_MODEL, SS);
ArduinoLEDMatrix matrix;
WiFiClient client;

char ssid[] = "YourWiFiName";
char pass[] = "YourWiFiPassword";
char serverIP[] = "192.168.1.XXX";
int serverPort = 5000;

void setup() {
  Serial.begin(9600);
  while (!Serial) { ; }
  
  matrix.begin();
  Wire.begin();
  SPI.begin();
  
  // Initialize camera
  myCAM.set_format(JPEG);
  myCAM.InitCAM();
  myCAM.set_bit(ARDUCHIP_TIM, VSYNC_LEVEL_MASK);
  myCAM.clear_fifo_flag();
  
  // Connect to WiFi
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("\nConnected!");
}

void loop() {
  // Capture image
  Serial.println("Capturing...");
  myCAM.flush_fifo();
  myCAM.clear_fifo_flag();
  myCAM.start_capture();
  
  while (!myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK)) { ; }
  
  uint32_t size = myCAM.read_fifo_length();
  Serial.print("Size: ");
  Serial.println(size);
  
  // Send to server and get classification
  if (client.connect(serverIP, serverPort)) {
    // Send image (simplified — see note below)
    String json = "{\"image\":\"placeholder\"}";
    
    client.println("POST /upload HTTP/1.1");
    client.println("Host: " + String(serverIP));
    client.println("Content-Type: application/json");
    client.print("Content-Length: ");
    client.println(json.length());
    client.println();
    client.println(json);
    
    // Read the response
    String response = "";
    while (client.available()) {
      char c = client.read();
      response += c;
    }
    
    Serial.println(response);
    
    // Light up LED based on classification
    // Different patterns for different object types
    if (response.indexOf("water_bottle") > 0 || 
        response.indexOf("cup") > 0) {
      // Liquid container: wave pattern
      for (int i = 0; i < 3; i++) {
        matrix.rect(0, i * 2, 8, 2, 1);
        delay(200);
        matrix.fill(0);
      }
    } else if (response.indexOf("mouse") > 0 || 
               response.indexOf("keyboard") > 0) {
      // Tech device: flash center
      matrix.rect(2, 2, 4, 4, 1);
      delay(500);
      matrix.fill(0);
    } else if (response.indexOf("person") > 0) {
      // Person: heart-like pattern (top half)
      matrix.rect(0, 0, 8, 4, 1);
      delay(500);
      matrix.fill(0);
    } else {
      // Default: full flash
      matrix.fill(1);
      delay(300);
      matrix.fill(0);
    }
    
    client.stop();
  }
  
  delay(5000);
}

Note: The full implementation would read the camera's FIFO buffer, base64-encode the JPEG data, and send it as a proper JSON payload. The Arducam library examples show how to read the FIFO. For this lesson, the placeholder approach lets us focus on the pipeline architecture.

Activity: Create Your Own Object → LED Mapping (10 min)

Now it's your turn. Modify the sketch to create your own LED patterns for different object categories. Think about:

What objects do you encounter at your desk every day?
What LED pattern would represent each one?
Can you make the pattern animate (like a spinning wheel or a pulsing light)?

Try these ideas:

Book/paper: A scrolling text effect (simulate reading)
Phone: A ringing pattern (alternating sides)
Food: A rainbow pattern
Plant: A slow breathing green pulse

Activity: End-to-End Demo (10 min)

Let's put it all together. Here's the complete flow:

Point the camera at an object
Click the mouse (or wait for the auto-capture loop)
Uno Q captures the image and sends it over WiFi
Laptop receives the image and runs MobileNet
MobileNet classifies the object (e.g., "water bottle, 95% confidence")
Python server sends the label to Ollama with a prompt: "Tell me an interesting fact about a water bottle"
Ollama generates a response
Python server sends both the label and the LLM response back to the Uno Q
Uno Q lights up the LED array with a pattern matching the object
Laptop screen displays the classification and the interesting fact

Try it with at least 5 different objects. Note which ones the model gets right and which it struggles with.

Reflection: What You Built (5 min)

Think about the journey:

Week 1: You learned what AI is and why your perspective matters
Week 2: You ran AI models locally and understood data sovereignty
Week 3: You connected hardware to AI over a network
Week 4: You built a multi-modal AI system that sees, thinks, and responds

You've built something that didn't exist before: a custom AI-powered device that you control, running entirely on your own hardware with your own data staying on your own network.

Discussion prompt: "What would you build next if you had unlimited time and parts? What real-world problem could a device like this solve?"

Take-Home

Check Your Understanding

What does SPI stand for and why is it used for the camera instead of USB?
Why do we send images from the Uno Q to the laptop for classification instead of running the model on the Uno Q itself?
What is a multi-modal AI pipeline? Give an example from this lesson.
How does the confidence score from MobileNet help you evaluate the model's output?
What are the three types of data flowing through your pipeline (input, intermediate, output)?

Assignment

Train the My Room app (from Week 2) on 3-5 objects from your own room, then use those custom categories in your Uno Q pipeline instead of MobileNet's 1,000 pre-trained categories
Add a new LED pattern for an object category not covered in class
Modify the LLM prompt to ask for different information (e.g., "What is this object used for?" instead of "Tell me an interesting fact")
Write a short reflection: How does having a physical AI device change your relationship with AI compared to using a chatbot on your phone?

Optional Supplemental Reading

Arducam OV2640 Datasheet
MobileNetV2 Paper — The architecture behind the vision model
TensorFlow Keras Applications — Other pre-trained models you can try
Multi-modal AI Explained

Next Steps

Capstone — Build Your Cyberdeck

Week 1: Discover

Week 2: Create

Capstone

Can You See Me Now? Adding Computer Vision to Your Arduino

Session Plan

Part 1 — Camera Setup: Your Arduino Learns to See (20 min)

Part 2 — Image Over WiFi: Sending Pictures to Your Laptop (20 min)

Part 3 — Vision Pipeline: Classifying What the Camera Sees (20 min)

Take-Home

Next Steps

Can You See Me Now? Adding Computer Vision to Your Arduino ​

Session Plan ​

Part 1 — Camera Setup: Your Arduino Learns to See (20 min) ​

Part 2 — Image Over WiFi: Sending Pictures to Your Laptop (20 min) ​

Part 3 — Vision Pipeline: Classifying What the Camera Sees (20 min) ​

Part 4 — Vision + Language + LEDs: The Complete Multi-Modal Pipeline (30 min) ​

Take-Home ​

Next Steps ​

Can You See Me Now? Adding Computer Vision to Your Arduino

Session Plan

Part 1 — Camera Setup: Your Arduino Learns to See (20 min)

Part 2 — Image Over WiFi: Sending Pictures to Your Laptop (20 min)

Part 3 — Vision Pipeline: Classifying What the Camera Sees (20 min)

Part 4 — Vision + Language + LEDs: The Complete Multi-Modal Pipeline (30 min)

Take-Home

Next Steps