Teachable Machine — Single Lighting Training Fails

Teachable Machine Image Classification — Training Checklist

1. Create classes (one per category you want to detect)
   - Minimum: 2 classes
   - Recommended: 3-10 classes
   - Avoid: more than 15 classes with fewer than 50 images each
   - Consider: add an "unknown/other" class with diverse non-target images

2. Add images per class
   - Minimum: 20 images per class (will overfit — demo only)
   - Recommended: 50-200 images per class
   - Best: 200+ images with systematically varied conditions
   - Critical: similar image counts per class (avoid 200 in one, 30 in another)

3. Image quality guidelines — the make-or-break factor
   - USE THE SAME DEVICE as deployment (phone camera ≠ DSLR ≠ laptop webcam)
   - Vary angles: front, side, top, 45-degree tilts, slight rotations
   - Vary lighting: bright daylight, dim indoor, overhead fluorescent, warm LED
   - Vary backgrounds: plain walls, cluttered desks, outdoors, dark surfaces
   - Vary distance: close-up, arm's length, across a table
   - Include edge cases: partially visible objects, slight blur, small size
   - Include negative examples in your "other" class: things that look similar but are not

4. Click Train Model
   - Training takes 30-120 seconds in the browser
   - Preview accuracy appears in the right panel — but this is training accuracy
   - Test with NEW images that were NOT in the training set
   - If accuracy is below 70%, add more diverse images before proceeding

5. Export Model
   - Click 'Export Model'
   - Select 'TensorFlow.js' tab
   - Choose 'Download' (not shareable link — those expire)
   - Download gives you: model.json + weights.bin files
   - Save these files — you cannot re-download later without retraining

Exported Model File Structure:

my_model/
  model.json          — Model architecture + metadata (class labels, input shape)
  weights.bin         — Trained weight values (may be split into multiple shards)

Key fields inside model.json:
  - modelTopology:       Neural network graph definition (layers, connections)
  - weightsManifest:     Paths to weight shard files with byte offsets and sizes
  - userDefinedMetadata: Contains your class labels array (the names you assigned)
  - format:              'layers-model' (the TensorFlow.js layers API format)

File sizes (typical for Teachable Machine export):
  - model.json:   50-150 KB
  - weights.bin:  2-5 MB (depends on number of classes)
  - Total:        ~3-6 MB before compression, ~1.5-3 MB with gzip

Deployment options:
  Option A: Download files → place in /public/models/ in your Next.js project
            Pros: fast, no external dependency, works offline, full control
            Cons: model updates require redeployment

  Option B: Use the shareable link directly (fetches from Google CDN)
            Pros: zero setup, instant sharing
            Cons: link expires, requires internet, slow in some regions

  Recommended: Option A for anything beyond a quick demo.

// app/page.jsx — Main page with dynamic import to avoid SSR of TF.js
import dynamic from 'next/dynamic';

// Dynamic import with ssr: false prevents the component from being
// rendered on the server, where browser APIs do not exist.
// The loading fallback is shown while the component JS downloads.
const ImageClassifier = dynamic(
  () => import('@/components/ImageClassifier'),
  {
    ssr: false,
    loading: () => (
      <div style={{ padding: '2rem', textAlign: 'center' }}>
        <p>Loading image classifier…</p>
        <p style={{ fontSize: '0.85rem', color: '#666' }}>
          Downloading model files (~3 MB)
        </p>
      </div>
    )
  }
);

export default function Home() {
  return (
    <main style={{ maxWidth: '800px', margin: '0 auto', padding: '2rem' }}>
      <h1>Image Classifier</h1>
      <p>
        Upload an image or use your webcam to classify objects.
        All processing happens in your browser — nothing is sent to a server.
      </p>
      <ImageClassifier />
    </main>
  );
}

// --- Installation ---
// npm install @tensorflow/tfjs @tensorflow/tfjs-backend-webgl
//
// Place your exported model files at:
//   public/models/my_model/model.json
//   public/models/my_model/weights.bin

'use client';

import { useState, useEffect, useRef, useCallback } from 'react';
import * as tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-backend-webgl';

const MODEL_URL = '/models/my_model/model.json';

export default function ImageClassifier() {
  const [model, setModel] = useState(null);
  const [labels, setLabels] = useState([]);
  const [loading, setLoading] = useState(true);
  const [predictions, setPredictions] = useState([]);
  const [error, setError] = useState(null);
  const fileInputRef = useRef(null);
  const imageRef = useRef(null);
  const modelRef = useRef(null);

  // Load model once on mount, dispose on unmount
  useEffect(() => {
    let mounted = true;

    async function init() {
      try {
        // Wait for TF.js backend to initialize
        await tf.ready();
        console.log('TF.js backend:', tf.getBackend());

        // Load the exported Teachable Machine model
        const loadedModel = await tf.loadLayersModel(MODEL_URL);

        // Warm up: first prediction compiles WebGL shaders (3-8s)
        // Do this now, behind the loading spinner, not on first user action
        const dummy = tf.zeros([1, 224, 224, 3]);
        const warmup = loadedModel.predict(dummy);
        warmup.dispose();
        dummy.dispose();

        // Extract class labels from model metadata
        // Teachable Machine stores labels in the model metadata
        let classLabels = [];
        try {
          const metadataUrl = MODEL_URL.replace('model.json', 'metadata.json');
          const metadataResponse = await fetch(metadataUrl);
          if (metadataResponse.ok) {
            const metadata = await metadataResponse.json();
            classLabels = metadata.labels || [];
          }
        } catch (metaErr) {
          console.warn('No metadata.json found, using generic labels');
          const outputShape = loadedModel.outputShape;
          const numClasses = outputShape[outputShape.length - 1];
          classLabels = Array.from(
            { length: numClasses },
            (_, i) => `Class ${i}`
          );
        }

        if (mounted) {
          setModel(loadedModel);
          modelRef.current = loadedModel;
          setLabels(classLabels);
          setLoading(false);
          console.log(
            `Model loaded. ${classLabels.length} classes: ${classLabels.join(', ')}`
          );
        } else {
          loadedModel.dispose();
        }
      } catch (err) {
        console.error('Model loading failed:', err);
        if (mounted) {
          setError(`Failed to load model: ${err.message}`);
          setLoading(false);
        }
      }
    }

    init();

    // Cleanup: dispose model on unmount to free GPU memory
    return () => {
      mounted = false;
      if (modelRef.current) {
        modelRef.current.dispose();
        console.log('Model disposed on unmount');
      }
    };
  }, []);

  // Classify an image element
  const classifyImage = useCallback(
    async (imgElement) => {
      if (!model || !imgElement) return;

      // All tensor ops wrapped in tf.tidy() for automatic memory cleanup
      const outputTensor = tf.tidy(() => {
        // Step 1: Read pixel data from the DOM image element
        const imageTensor = tf.browser.fromPixels(imgElement);

        // Step 2: Resize to 224x224 (MobileNet's expected input size)
        const resized = tf.image.resizeBilinear(imageTensor, [224, 224]);

        // Step 3: Convert to float and normalize to [-1, 1]
        // MobileNet expects this range, NOT [0, 1]
        const normalized = resized.toFloat().div(127.5).sub(1.0);

        // Step 4: Add batch dimension [224,224,3] → [1,224,224,3]
        const batched = normalized.expandDims(0);

        // Step 5: Run inference
        return model.predict(batched);
      });

      // Extract probabilities from GPU tensor to CPU array
      const probabilities = await outputTensor.data();
      outputTensor.dispose(); // Manual dispose for async result

      // Sort predictions by confidence, highest first
      const sorted = Array.from(probabilities)
        .map((prob, i) => ({
          label: labels[i] || `Class ${i}`,
          confidence: (prob * 100).toFixed(1),
        }))
        .sort((a, b) => b.confidence - a.confidence);

      setPredictions(sorted);
    },
    [model, labels]
  );

  // Handle file upload
  const handleFileUpload = (event) => {
    const file = event.target.files[0];
    if (!file) return;

    const reader = new FileReader();
    reader.onload = (e) => {
      const img = imageRef.current;
      img.src = e.target.result;
      // Wait for image to fully load before classifying
      img.onload = () => classifyImage(img);
    };
    reader.readAsDataURL(file);
  };

  if (loading) {
    return (
      <div style={{ padding: '2rem', textAlign: 'center' }}>
        <p>Initializing model…</p>
        <p style={{ fontSize: '0.85rem', color: '#666' }}>
          This takes a few seconds on first load
        </p>
      </div>
    );
  }
  if (error) return <p style={{ color: 'red' }}>{error}</p>;

  return (
    <div>
      <input
        ref={fileInputRef}
        type="file"
        accept="image/*"
        onChange={handleFileUpload}
        style={{ marginBottom: '1rem' }}
      />

      <img
        ref={imageRef}
        alt="Upload preview"
        style={{
          maxWidth: '300px',
          marginTop: '1rem',
          display: 'block',
          borderRadius: '8px',
        }}
      />

      {predictions.length > 0 && (
        <div style={{ marginTop: '1rem' }}>
          <h3>Predictions</h3>
          {predictions.map((p, i) => (
            <div key={i} style={{ marginBottom: '0.5rem' }}>
              <strong>{p.label}</strong>: {p.confidence}%
              <div
                style={{
                  height: '8px',
                  background: '#e0e0e0',
                  borderRadius: '4px',
                  marginTop: '4px',
                }}
              >
                <div
                  style={{
                    width: `${p.confidence}%`,
                    height: '100%',
                    background: i === 0 ? '#4caf50' : '#90caf9',
                    borderRadius: '4px',
                    transition: 'width 0.3s ease',
                  }}
                />
              </div>
            </div>
          ))}
        </div>
      )}
    </div>
  );
}

'use client';

import { useState, useEffect, useRef, useCallback } from 'react';
import * as tf from '@tensorflow/tfjs';

export default function WebcamClassifier({ model, labels }) {
  const videoRef = useRef(null);
  const [predictions, setPredictions] = useState([]);
  const [isStreaming, setIsStreaming] = useState(false);
  const animationRef = useRef(null);
  const isProcessingRef = useRef(false);

  const startWebcam = async () => {
    try {
      const stream = await navigator.mediaDevices.getUserMedia({
        video: {
          width: { ideal: 224 },
          height: { ideal: 224 },
          facingMode: 'environment', // Rear camera on mobile
        },
      });
      videoRef.current.srcObject = stream;
      await videoRef.current.play();
      setIsStreaming(true);
      classifyFrame();
    } catch (err) {
      console.error('Webcam access denied or unavailable:', err);
      // Common causes: no camera, permission denied, HTTP (not HTTPS)
    }
  };

  const stopWebcam = () => {
    if (animationRef.current) {
      cancelAnimationFrame(animationRef.current);
      animationRef.current = null;
    }
    if (videoRef.current?.srcObject) {
      videoRef.current.srcObject.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
    }
    setIsStreaming(false);
    setPredictions([]);
  };

  const classifyFrame = useCallback(() => {
    // Skip this frame if the previous inference is still running
    // This prevents GPU memory buildup from queued predictions
    if (!model || !videoRef.current || isProcessingRef.current) {
      animationRef.current = requestAnimationFrame(classifyFrame);
      return;
    }

    // Check that video is actually producing frames
    if (
      videoRef.current.readyState < 2 ||
      videoRef.current.videoWidth === 0
    ) {
      animationRef.current = requestAnimationFrame(classifyFrame);
      return;
    }

    isProcessingRef.current = true;

    // Synchronous tensor ops wrapped in tf.tidy for auto-cleanup
    const result = tf.tidy(() => {
      const frame = tf.browser.fromPixels(videoRef.current);
      const resized = tf.image.resizeBilinear(frame, [224, 224]);
      const normalized = resized.toFloat().div(127.5).sub(1.0);
      const batched = normalized.expandDims(0);
      return model.predict(batched);
    });

    // Async data extraction — must dispose manually
    result
      .data()
      .then((probabilities) => {
        const sorted = Array.from(probabilities)
          .map((prob, i) => ({
            label: labels[i] || `Class ${i}`,
            confidence: (prob * 100).toFixed(1),
          }))
          .sort((a, b) => b.confidence - a.confidence)
          .slice(0, 3);

        setPredictions(sorted);
      })
      .finally(() => {
        result.dispose();
        isProcessingRef.current = false;
        animationRef.current = requestAnimationFrame(classifyFrame);
      });
  }, [model, labels]);

  // Cleanup on unmount: stop webcam, cancel animation frame
  useEffect(() => {
    return () => stopWebcam();
  }, []);

  return (
    <div>
      <button onClick={isStreaming ? stopWebcam : startWebcam}>
        {isStreaming ? 'Stop Camera' : 'Start Camera'}
      </button>

      <video
        ref={videoRef}
        style={{
          width: '300px',
          marginTop: '1rem',
          display: isStreaming ? 'block' : 'none',
          borderRadius: '8px',
        }}
        playsInline
        muted
      />

      {predictions.length > 0 && isStreaming && (
        <div style={{ marginTop: '1rem' }}>
          {predictions.map((p, i) => (
            <div key={i} style={{ marginBottom: '0.25rem' }}>
              <strong>{p.label}</strong>: {p.confidence}%
            </div>
          ))}
        </div>
      )}
    </div>
  );
}

// next.config.mjs — Optimize model file serving
/** @type {import('next').NextConfig} */
const nextConfig = {
  // Enable gzip compression for all static assets
  compress: true,

  // Custom headers for model files — cache aggressively
  async headers() {
    return [
      {
        // Match all files under /models/
        source: '/models/:path*',
        headers: [
          {
            // Cache for 1 year. Model files do not change after deployment.
            // When you update the model, change the directory name (v1 → v2).
            key: 'Cache-Control',
            value: 'public, max-age=31536000, immutable',
          },
        ],
      },
    ];
  },
};

export default nextConfig;

// --- File placement ---
// public/
//   models/
//     my_model/
//       model.json      ← Model architecture and metadata
//       weights.bin     ← Trained weight values
//       metadata.json   ← Class labels (if exported separately)
//
// These files are served at:
//   /models/my_model/model.json
//   /models/my_model/weights.bin
//   /models/my_model/metadata.json
//
// --- Model versioning ---
// When updating the model, use a new directory:
//   /models/my_model_v2/model.json
// Update MODEL_URL in your component to point to the new version.
// Old cached versions will not interfere because the path changed.

'use client';

import { useState, useCallback } from 'react';
import * as tf from '@tensorflow/tfjs';

// Minimum confidence to show a definitive prediction.
// Below this threshold, the prediction is flagged as uncertain.
// Tune this on your held-out test set — not by intuition.
const CONFIDENCE_THRESHOLD = 70;

export default function ClassifierWithFallback({
  model,
  labels,
  imageElement,
}) {
  const [result, setResult] = useState(null);

  const classify = useCallback(async () => {
    if (!model || !imageElement) return;

    // Verify image is actually loaded and visible
    if (
      !imageElement.naturalWidth ||
      !imageElement.naturalHeight
    ) {
      console.warn('Image not loaded yet — skipping classification');
      return;
    }

    const prediction = tf.tidy(() => {
      const tensor = tf.browser
        .fromPixels(imageElement)
        .resizeBilinear([224, 224])
        .toFloat()
        .div(127.5)
        .sub(1.0)
        .expandDims(0);
      return model.predict(tensor);
    });

    const probabilities = await prediction.data();
    prediction.dispose();

    const sorted = Array.from(probabilities)
      .map((prob, i) => ({
        label: labels[i] || `Class ${i}`,
        confidence: prob * 100,
      }))
      .sort((a, b) => b.confidence - a.confidence);

    const topPrediction = sorted[0];
    const secondPrediction = sorted[1];

    // Check confidence AND gap to second prediction
    // A 72% prediction with the second at 71% is very different
    // from a 72% prediction with the second at 5%
    const confidenceGap = topPrediction.confidence - (secondPrediction?.confidence || 0);

    if (
      topPrediction.confidence < CONFIDENCE_THRESHOLD ||
      confidenceGap < 15 // Too close between top two classes
    ) {
      setResult({
        status: 'uncertain',
        message: `Low confidence (${topPrediction.confidence.toFixed(1)}%) or ambiguous classification. Please review manually.`,
        topGuesses: sorted.slice(0, 3),
      });
    } else {
      setResult({
        status: 'confident',
        label: topPrediction.label,
        confidence: topPrediction.confidence.toFixed(1),
        allPredictions: sorted,
      });
    }
  }, [model, labels, imageElement]);

  return { result, classify };
}

Teachable Machine + Next.js Deployment Checklist

=== Pre-deployment Validation ===
  [ ] Model files downloaded and placed in /public/models/
  [ ] Shareable link is NOT used in production code
  [ ] Model tested on at least 3 device types (desktop Chrome, iOS Safari, Android Chrome)
  [ ] Accuracy validated on held-out test images that were NOT in the training set
  [ ] Confidence threshold tested and tuned on the held-out set
  [ ] WebGL backend verified on all target browsers (check tf.getBackend())
  [ ] Model tested with edge case inputs: blank images, rotated, very dark, very bright

=== Code Quality ===
  [ ] dynamic(() => import(...), { ssr: false }) used for all TF.js components
  [ ] 'use client' directive present on every file that imports TensorFlow.js
  [ ] tf.tidy() wraps ALL synchronous tensor operations
  [ ] Async tensor results (.data()) followed by .dispose() in .finally()
  [ ] useEffect cleanup disposes model on unmount via modelRef
  [ ] Model warmup (dummy prediction) runs during component mount behind loading UI
  [ ] Webcam streams stopped and tracks released on component unmount

=== Performance ===
  [ ] Gzip compression enabled for model files (next.config.mjs compress: true)
  [ ] Cache-Control: public, max-age=31536000, immutable set on /models/* paths
  [ ] Loading indicator with descriptive text shown during model download
  [ ] First user interaction triggers instant prediction (warmup already complete)
  [ ] tf.memory().numTensors monitored in dev — stays constant across predictions

=== Monitoring and Error Handling ===
  [ ] Console logs backend type on init (webgl vs cpu vs wasm)
  [ ] Error boundary or try/catch wraps model loading
  [ ] Fallback UI shown for devices without WebGL support
  [ ] HTTPS enabled (required for webcam access in production)
  [ ] Image onload event gates classification — never classify a half-loaded image