Skip to main content

​
Rerun Lakehouse Visualizer — Flickr8k Demo

Interactive multimodal visualization of the Flickr8k dataset stored in the Auraison lakehouse (s3://landing/flickr-8k/), powered by the Rerun SDK. What this demo shows:
  • Loading image + caption pairs directly from MinIO S3 via boto3
  • Logging them to Rerun with a sequential timeline (image_idx)
  • Saving a portable .rrd recording that can be shared or embedded
  • Embedding the Rerun web viewer in a Mintlify MDX page via <iframe>
Prerequisites: 
cd data-plane
# External TrueNAS infra — set .env with MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY
# Or start local infra:
# docker compose --profile local-infra up -d

import os, sys
# Ensure data-plane root is on the path
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath('.')), ''))

S3_ENDPOINT = os.getenv('MINIO_ENDPOINT', 'http://localhost:29000')
S3_KEY      = os.getenv('MINIO_ACCESS_KEY', 'minio')
S3_SECRET   = os.getenv('MINIO_SECRET_KEY', 'minio123')

BUCKET   = 'landing'
PREFIX   = 'flickr-8k'
N_IMAGES = 20          # images to stream into Rerun
RRD_PATH = '/tmp/flickr8k_demo.rrd'

print(f'MinIO endpoint : {S3_ENDPOINT}')
print(f'Dataset prefix : s3://{BUCKET}/{PREFIX}/')
print(f'Images to load : {N_IMAGES}')
print(f'RRD output     : {RRD_PATH}')

MinIO endpoint : http://192.168.1.26:9000
Dataset prefix : s3://landing/flickr-8k/
Images to load : 20
RRD output     : /tmp/flickr8k_demo.rrd

​
Step 1 — Load Captions from S3

Flickr8k caption file format: CSV with image,caption columns. Each image has 5 reference captions from different annotators. 
import boto3
from botocore.config import Config
from collections import defaultdict

s3 = boto3.client(
    's3',
    endpoint_url=S3_ENDPOINT,
    aws_access_key_id=S3_KEY,
    aws_secret_access_key=S3_SECRET,
    config=Config(signature_version='s3v4'),
)

# Fetch captions index
obj = s3.get_object(Bucket=BUCKET, Key=f'{PREFIX}/text/captions.txt')
lines = obj['Body'].read().decode('utf-8').splitlines()

captions = defaultdict(list)
for line in lines[1:]:          # skip header
    parts = line.split(',', 1)
    if len(parts) == 2:
        fname, cap = parts
        captions[fname.strip()].append(cap.strip())
    if len(captions) >= N_IMAGES:
        break

filenames = list(captions.keys())[:N_IMAGES]

print(f'Caption file: {len(lines)-1} entries total')
print(f'Unique images selected: {len(filenames)}')
print(f'\nSample — {filenames[0]}:')
for i, cap in enumerate(captions[filenames[0]]):
    print(f'  {i+1}. {cap}')

Caption file: 40455 entries total
Unique images selected: 20

Sample — 1000268201_693b08cb0e.jpg:
  1. A child in a pink dress is climbing up a set of stairs in an entry way .
  2. A girl going into a wooden building .
  3. A little girl climbing into a wooden playhouse .
  4. A little girl climbing the stairs to her playhouse .
  5. A little girl in a pink dress going into a wooden cabin .

​
Step 2 — Log Images + Captions to Rerun

We initialise a Rerun recording stream and save it to a .rrd file. Each image is logged as rr.EncodedImage (JPEG bytes, no decode needed) and each set of captions as rr.TextDocument with Markdown formatting. Timeline: image_idx — a sequence timeline, one step per image. 
import rerun as rr
import rerun.blueprint as rrb

# Initialise recording (headless — no viewer spawned)
rr.init('auraison/flickr8k', spawn=False)
rr.save(RRD_PATH)

# Blueprint: image on left (3/5 width), captions on right (2/5)
blueprint = rrb.Horizontal(
    rrb.Spatial2DView(name='Image', origin='/image'),
    rrb.TextDocumentView(name='Captions', origin='/captions'),
    column_shares=[3, 2],
)
rr.send_blueprint(blueprint)

print(f'Streaming {len(filenames)} images → {RRD_PATH} …\n')

for idx, fname in enumerate(filenames):
    img_bytes = s3.get_object(
        Bucket=BUCKET, Key=f'{PREFIX}/images/{fname}'
    )['Body'].read()

    rr.set_time('image_idx', sequence=idx)

    # JPEG-encoded image — no PIL decode needed
    rr.log('/image', rr.EncodedImage(contents=img_bytes, media_type='image/jpeg'))

    # All 5 captions as a Markdown document
    caption_md = f'**{fname}**\n\n' + '\n\n'.join(
        f'{i+1}. {c}' for i, c in enumerate(captions[fname])
    )
    rr.log('/captions', rr.TextDocument(caption_md, media_type=rr.MediaType.MARKDOWN))

    size_kb = len(img_bytes) / 1024
    print(f'  [{idx+1:2d}/{len(filenames)}]  {fname}  ({size_kb:.0f} KiB)')

print(f'\n✓ Recording saved to {RRD_PATH}')
import os
rrd_size_kb = os.path.getsize(RRD_PATH) / 1024
print(f'  File size: {rrd_size_kb:.0f} KiB')

Streaming 20 images → /tmp/flickr8k_demo.rrd …

  [ 1/20]  1000268201_693b08cb0e.jpg  (195 KiB)
  [ 2/20]  1001773457_577c3a7d70.jpg  (139 KiB)
  [ 3/20]  1002674143_1b742ab4b8.jpg  (156 KiB)
  [ 4/20]  1003163366_44323f5815.jpg  (150 KiB)
  [ 5/20]  1007129816_e794419615.jpg  (118 KiB)
  [ 6/20]  1007320043_627395c3d8.jpg  (175 KiB)
  [ 7/20]  1009434119_febe49276a.jpg  (166 KiB)
  [ 8/20]  1012212859_01547e3f17.jpg  (124 KiB)
  [ 9/20]  1015118661_980735411b.jpg  (157 KiB)
  [10/20]  1015584366_dfcec3c85a.jpg  (145 KiB)
  [11/20]  101654506_8eb26cfb60.jpg  (99 KiB)
  [12/20]  101669240_b2d3e7f17b.jpg  (161 KiB)
  [13/20]  1016887272_03199f49c4.jpg  (115 KiB)
  [14/20]  1019077836_6fc9b15408.jpg  (251 KiB)
  [15/20]  1019604187_d087bf9a5f.jpg  (110 KiB)
  [16/20]  1020651753_06077ec457.jpg  (161 KiB)
  [17/20]  1022454332_6af2c1449a.jpg  (128 KiB)
  [18/20]  1022454428_b6b660a67b.jpg  (110 KiB)
  [19/20]  1022975728_75515238d8.jpg  (101 KiB)
  [20/20]  102351840_323e3de834.jpg  (48 KiB)

✓ Recording saved to /tmp/flickr8k_demo.rrd
  File size: 2175 KiB

​
Step 3 — Open in the Rerun Web Viewer

To view the recording locally, run: 
# Serve via gRPC + open browser automatically
uv run python scripts/rerun_demo.py --count 20

# Or open an existing .rrd file with the native viewer
rerun /tmp/flickr8k_demo.rrd
The recording can also be embedded in any web page using the hosted Rerun viewer at https://app.rerun.io — see the Mintlify Embed section below.

​
What Rerun shows

PanelContent
Left (Spatial 2D)JPEG image — pan, zoom, inspect pixels
Right (Text Document)5 reference captions in Markdown
Bottom timelineimage_idx — scrub to navigate images

​
Step 4 — Mintlify Embed (iframe)

The Rerun web viewer at https://app.rerun.io can load any publicly hosted .rrd file via its url query parameter. The embed below uses Rerun’s own ARKit Scenes example to validate that the Mintlify <iframe> component renders correctly: 
<iframe
  src="https://app.rerun.io/version/0.29.2/index.html?url=https://app.rerun.io/version/0.29.2/examples/arkit_scenes.rrd"
  width="100%"
  height="600px"
  style={{ border: 'none', borderRadius: '8px' }}
/>
To embed the Flickr8k recording instead, upload flickr8k_demo.rrd to a public URL (e.g. Cloudflare R2, GitHub Releases, or MinIO with public bucket policy) and substitute the url parameter.

​
Live Embed Test

The iframe below loads the Rerun web viewer with the ARKit Scenes example recording to verify that the Mintlify <iframe> component renders correctly: 
print('=== Rerun Lakehouse Visualizer — Summary ===')
print(f'Dataset        : Flickr8k  (landing/flickr-8k/)')
print(f'Images logged  : {len(filenames)}')
print(f'Captions/image : 5 reference captions each')
print(f'Recording file : {RRD_PATH}  ({rrd_size_kb:.0f} KiB)')
print()
print('Entity tree logged to Rerun:')
print('  /image    → rr.EncodedImage (JPEG)')
print('  /captions → rr.TextDocument (Markdown)')
print()
print('Timeline: image_idx (sequence, 0 …', len(filenames)-1, ')')
print()
print('To serve live:')
print('  uv run python scripts/rerun_demo.py --count 20')
print('  → http://localhost:9090')

=== Rerun Lakehouse Visualizer — Summary ===
Dataset        : Flickr8k  (landing/flickr-8k/)
Images logged  : 20
Captions/image : 5 reference captions each
Recording file : /tmp/flickr8k_demo.rrd  (2175 KiB)

Entity tree logged to Rerun:
  /image    → rr.EncodedImage (JPEG)
  /captions → rr.TextDocument (Markdown)

Timeline: image_idx (sequence, 0 … 19 )

To serve live:
  uv run python scripts/rerun_demo.py --count 20
  → http://localhost:9090
Edit this page on GitHub or file an issue.