EST. 2012 NYC · ACQUIRED BY TWITTER JULY 2014

PIONEERING MULTIMODAL PERCEPTION Turning Raw Web Media Into Natural Language

Co-founded in 2012 by Clément Farabet and Louis-Alexandre Etezad-Heydari, MadBits pioneered web-scale visual intelligence—engineering early breakthrough convolutional neural networks coupled with recurrent sequence decoders to automatically parse, classify, and transcribe raw web media into descriptive text and semantic indices.

EXPLORE NEURAL PIPELINE → READ THE CHRONOLOGY

// MADBITS.COM MISSION ARCHIVE (2013)

“Over this past year, we’ve built visual intelligence technology that automatically understands, organizes and extracts relevant information from raw media.”

— MadBits Co-Founders, July 2014

MADBITS TELEMETRY // CUDA 5.5 // TORCH7

PRIMARY ARCHITECTURE

Spatial ConvNet + Recurrent LSTM

TRAINING FRAMEWORK

Torch7 (LuaJIT / libtorch / CUDA)

TARGET HARDWARE

NVIDIA Tesla K20X / Kepler Titan

ACQUISITION IMPACT

Foundation of Twitter Cortex (2014)

END-TO-END MULTIMODAL FORWARD PASS

ZERO MANUAL TAGS REQUIRED PIXEL-TO-LANGUAGE INFERENCE

// INTERACTIVE LABORATORY

Pixels to Language: The 2012 Neural Pipeline

Experience how MadBits unified spatial convolutional feature extraction with sequential LSTM language decoding to transcribe raw visual media in real time.

SAMPLE INPUT:

Upload Custom Media

STAGE 01 RAW INPUT TENSOR 3×256×256 FP32

Channel Range: [0.0, 1.0] Normalized

Input Modality: Unstructured Pixels

Batch Allocation: 1 × 3 × 256 × 256

STAGE 02 CONVOLUTIONAL FEATURE MAPS Torch7 nn.SpatialConv

SELECT ACTIVE FEATURE LAYER:

MAP 01 · HORIZ_GRADIENT

MAP 02 · VERT_GRADIENT

MAP 03 · LAPLACIAN_EDGE

MAP 04 · SALIENCY_HEATMAP

✦ Multi-scale spatial hierarchies extracted without manual descriptors.

STAGE 03 SPATIAL POOLING & LATENT VECTOR 512-DIM VECTOR

Global spatial average pooling compresses high-resolution feature maps into a semantic visual embedding vector z ∈ ℝ⁵¹², used to initialize the recurrent hidden state h₀.

LATENT EMBEDDING PROJECTION [0...511] ||z||₂ = 14.82

DETECTED SEMANTIC ACTIVATIONS:

#urban-transit #vehicle #daylight #street-level #crosswalk

STAGE 04 LSTM RECURRENT TEXT DECODER BEAM SEARCH (B=3)

madbits:decoder> INFERENCE COMPLETED

A yellow taxi cab driving down a city street with pedestrians.

SOFTMAX PROBABILITIES AT STEP 04: Entropy: 0.41 bits

"cab"

86.2%

"car"

8.4%

"automobile"

3.1%

"vehicle"

2.3%

// HISTORICAL CHRONOLOGY

The MadBits Journey: 2012 to 2014 & Beyond

How two NYU computational vision and perception researchers built the future of visual intelligence and laid the foundation for modern multimodal systems.

2012

THE GENESIS · NEW YORK CITY

Roots at NYU & The Birth of Torch7

MadBits was founded in late 2012 in New York City by Clément Farabet and Louis-Alexandre Etezad-Heydari. Both had emerged from New York University’s computational vision and neural science community—Farabet conducting deep learning research under AI pioneer Yann LeCun at the Courant Institute, and Etezad-Heydari researching human and computational visual perception alongside Laurence Maloney and Eero Simoncelli.

Clément had already co-developed Torch7—the LuaJIT scientific computing framework that powered deep learning before the creation of TensorFlow or PyTorch—alongside Ronan Collobert and Koray Kavukcuoglu. While the broader tech world was still fixated on manual feature engineering, Farabet and Etezad-Heydari recognized that neural networks running on NVIDIA GPUs were about to overturn all visual computing.

The Zuckerberg Call: As chronicled in Cade Metz’s book Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World, Mark Zuckerberg personally called Clément Farabet in late 2013 to recruit him as a foundational researcher for Facebook’s nascent artificial intelligence lab (FAIR). Farabet declined the Silicon Valley giant’s offer, choosing instead to remain committed to MadBits and pursue his independent vision.

2013

THE BREAKTHROUGH · MULTIMODAL PERCEPTION

Coupling ConvNets with Recurrent Sequence Models: Words from Raw Pixels

In 2013, the web was drowning in visual media, but search engines were blind: they could only index images if humans had manually tagged them with text keywords. MadBits inverted this equation.

The team developed an end-to-end architecture that joined multi-scale Convolutional Neural Networks (CNNs) for hierarchical visual feature extraction with Long Short-Term Memory (LSTM) recurrent models for sequence transcription and language generation. By feeding high-level spatial visual vectors directly into recurrent hidden states, MadBits achieved automatic image categorization, dense visual search, and natural language description prototypes across unannotated web imagery—preceding the late-2014 academic surge in image captioning.

code.madbits.com: During its operational years, MadBits was also a vital educational pillar for the AI community. The team maintained the open-access tutorial and documentation wiki at code.madbits.com, training an entire generation of early deep learning researchers on training neural networks with Torch.

JULY 2014

THE EXIT · SAN FRANCISCO & NYC

Acquired by Twitter to Power Real-Time Visual Media

On July 29, 2014, tech media worldwide reported that Twitter had acquired MadBits. With hundreds of millions of photos and video clips flowing through Twitter feeds every day without text descriptors, Twitter desperately needed deep learning visual intelligence to understand its media stream.

The MadBits acquisition marked Twitter’s first major leap into modern artificial intelligence. The MadBits team, technology stack, and neural algorithms were brought inside Twitter to automate media categorization, visual search, and real-time content filtering.

TechCrunch (July 2014)

Reported on Twitter’s acquisition of MadBits, highlighting its deep learning technology developed to automatically understand, organize, and extract relevant data from raw media without manual tags.

The Wall Street Journal (July 2014)

Noted Twitter’s entry into deep learning through MadBits, comparing the move to major industry pushes by Google and Facebook to parse visual media at web scale.

2014–TODAY

THE LEGACY · TWITTER CORTEX TO MODERN MULTIMODAL AI

Founding Twitter Cortex & The Seed of Modern VLMs

Immediately following the acquisition, Twitter Cortex was founded by Clément Farabet, Louis-Alexandre Etezad-Heydari, and Kevin Quennesson (who now runs braid.health) as Twitter’s central machine learning and AI organization. Cortex scaled deep learning infrastructure across the entire company, powering tweet ranking, visual recommendation graphs, media moderation, and real-time streaming intelligence.

Looking back, the CNN + LSTM paradigm pioneered by MadBits was the direct conceptual ancestor of today’s Vision-Language Models (VLMs), contrastive vision encoders, and multimodal generative transformers. What MadBits began with Torch7 and Kepler GPUs in 2012 proved that perception and language could be learned jointly from raw data.

// DEEP LEARNING ARCHAEOLOGY

The CNN + LSTM Multimodal Architecture

A granular breakdown of the mathematical and algorithmic components that enabled MadBits to bridge visual geometry and natural syntax.

MODULE A

Spatial Feature Extraction (CNN)

Spatial Convolution

The visual backbone utilized deep 2D spatial convolutions with non-saturating activation functions and spatial max-pooling:

y[c, i, j] = f( Σ_k Σ_(u,v) W[c, k, u, v] · x[k, i+u, j+v] + b[c] )

Multi-Scale Filters: Early 7×7 and 5×5 receptive fields to capture low-frequency illumination and edge primitives, transitioning into 3×3 receptive fields for dense semantic patterns.
Invariance to Perturbations: Subsampling and spatial pooling layers provided translational and scale invariance across diverse camera angles and image compressions.
CUDA Acceleration: Hand-optimized CUDA kernels in Torch7 allowing parallel matrix multiplication on GPU streaming multiprocessors.

MODULE B

Sequential Language Decoding (LSTM)

Long Short-Term Memory

To translate the global visual vector z into fluent language without suffering from vanishing gradients, MadBits deployed gated recurrent LSTM cells:

i_t = σ( W_xi · x_t + W_hi · h_{t-1} + b_i )
f_t = σ( W_xf · x_t + W_hf · h_{t-1} + b_f )
c_t = f_t ⊙ c_{t-1} + i_t ⊙ tanh( W_xc · x_t + W_hc · h_{t-1} + b_c )
h_t = σ( W_xo · x_t + W_ho · h_{t-1} + b_o ) ⊙ tanh( c_t )

Visual State Conditioning: The pooled convolutional vector z ∈ ℝ⁵¹² conditioned either the initial hidden state h₀ = tanh(W_z · z) or entered as an auxiliary input at every recurrent time step.
Beam Search Decoding: Maintained a top-$k$ beam of partial sentence hypotheses, maximizing cumulative log-probability over vocabulary tokens.
End-to-End Joint Loss: Cross-entropy loss on target tokens backpropagated through recurrent time steps directly into convolutional spatial filters.

// ARCHITECTURE SIMULATION & TORCH7 RECONSTRUCTION

The 2012 Torch7 / Lua Environment

Interactive Lua architecture models and simulated GPU execution logs reflecting MadBits’ multimodal pipeline design.

-- =========================================================================
-- MadBits Multimodal Neural Architecture (2012-2013 Reconstruction)
-- Framework: Torch7 / LuaJIT + cutorch + cunn
-- Co-founders: Clement Farabet & Louis-Alexandre Etezad-Heydari
-- =========================================================================

require 'torch'
require 'nn'
require 'cutorch'
require 'cunn'

-- 1. Construct Spatial Convolutional Feature Extractor
local convnet = nn.Sequential()
convnet:add(nn.SpatialConvolution(3, 64, 7, 7, 2, 2, 3, 3)) -- 128x128
convnet:add(nn.ReLU(true))
convnet:add(nn.SpatialMaxPooling(3, 3, 2, 2))             -- 64x64

convnet:add(nn.SpatialConvolution(64, 192, 5, 5, 1, 1, 2, 2))
convnet:add(nn.ReLU(true))
convnet:add(nn.SpatialMaxPooling(3, 3, 2, 2))             -- 32x32

convnet:add(nn.SpatialConvolution(192, 384, 3, 3, 1, 1, 1, 1))
convnet:add(nn.ReLU(true))
convnet:add(nn.SpatialConvolution(384, 512, 3, 3, 1, 1, 1, 1))
convnet:add(nn.ReLU(true))
convnet:add(nn.SpatialAdaptiveMaxPooling(1, 1))           -- 512-dim bottleneck
convnet:add(nn.View(512))

-- 2. Recurrent LSTM Language Decoder Module (Prototype sequence layer)
local lstm = nn.Sequential()
lstm:add(nn.LookupTable(vocab_size, 256))                -- Word Embeddings
lstm:add(nn.LSTM(256 + 512, 512))                        -- Conditioned on image vector
lstm:add(nn.Linear(512, vocab_size))
lstm:add(nn.LogSoftMax())

-- Ship tensors to CUDA device 1
convnet:cuda()
lstm:cuda()
print('[MadBits] Multimodal Vision-Language pipeline initialized in GPU VRAM')

-- =========================================================================
-- Joint Multimodal Loss Backpropagation (Torch7 optim.sgd)
-- =========================================================================

local criterion = nn.ClassNLLCriterion():cuda()
local optimState = { learningRate = 1e-3, momentum = 0.9, weightDecay = 5e-4 }

for epoch = 1, opt.max_epochs do
  for batch in dataloader:iter() do
    -- 1. Forward raw images through Convolutional layers
    local visual_feats = convnet:forward(batch.images:cuda())
    
    -- 2. Recurrently unroll LSTM over sequence time steps
    local loss = 0
    for t = 1, batch.seq_length do
      local pred = lstm:forward({batch.tokens[t]:cuda(), visual_feats})
      loss = loss + criterion:forward(pred, batch.targets[t]:cuda())
    end

    -- 3. Backpropagate error from language loss directly into ConvNet weights
    lstm:backward(...)
    convnet:backward(...)
  end
end

[SIMULATION LOG // TORCH7 KEPLER RECONSTRUCTION]
[INFO 2013-04-18 10:24:02] Initializing CUDA Context on NVIDIA Tesla K20X (Device 0)
[INFO 2013-04-18 10:24:02] Compute Capability: 3.5 | SM Count: 14 | Total VRAM: 5760 MB
[INFO 2013-04-18 10:24:03] Loaded cutorch v1.0 & cunn v1.0 (Torch7 distribution)
[INFO 2013-04-18 10:24:04] Allocating ConvNet weight tensor: 14,248,320 FP32 params (54.3 MB)
[INFO 2013-04-18 10:24:04] Allocating recurrent sequence model weights: 8,388,608 FP32 params (32.0 MB)
[INFO 2013-04-18 10:24:05] Memory Pool Initialized: 486.2 MB active
[INFO 2013-04-18 10:24:06] Forward pass warmup completed: 41.8 ms per 256x256 image
[SUCCESS] Pipeline ready for web-scale streaming inference on madbits.com

// ARCHIVE & IMPACT

From MadBits to Twitter Cortex & Beyond

How an early 2012 conviction paved the runway for Twitter’s core machine learning platform and today’s multimodal AI ecosystem.

The Spark for Twitter Cortex

Following Twitter’s acquisition of MadBits in July 2014, Twitter Cortex was founded by Clément Farabet, Louis-Alexandre Etezad-Heydari, and Kevin Quennesson (who now runs braid.health). Cortex became the foundational AI research and systems engineering organization inside Twitter, scaling deep neural networks to rank real-time timelines, discover breaking topics, detect media policy violations, and parse live streaming video.

The Shift to End-to-End Multimodal

Before MadBits, computer vision and natural language processing were entirely siloed disciplines. Vision engineers calculated SIFT or HOG features; NLP engineers parsed n-grams. MadBits demonstrated that visual feature representations could be aligned directly with recurrent language semantics in a unified differentiable computational graph.

Open Infrastructure Pioneers

Through their leadership in Torch7 and the educational resources distributed via code.madbits.com, the MadBits founders contributed critically to the open source deep learning foundation that eventually evolved into modern scientific computing and PyTorch.

HISTORICAL INTEGRITY & PRIMARY SOURCES

MadBits was incorporated in 2012 in New York City by Clément Farabet and Louis-Alexandre Etezad-Heydari. The acquisition of MadBits by Twitter was completed on July 29, 2014. Key references include TechCrunch, The Wall Street Journal, Cade Metz’s book Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World (Dutton, 2021), and the personal chronicles on clmt.ai.