RadiometricCamera.jl
RadiometricCamera calibrates a camera radiometrically, by inverse ray tracing in sensor × aperture space. A geometric camera model π⁻¹ — the calibration of a feature-based pipeline (BlobBoards, the central-generic spline model of Schöps et al.'s pipeline, Kalibr, a Kannala–Brandt or OpenCV fit, any VisualGeometryToolkitCore lens) — is refined on photographs of a printed board by rendering the board through the full radiometric camera (chief rays, finite aperture, focus surface, vignetting, lighting, the board's SVBSDF and warping) and minimising the photometric residual jointly over all parameters. The code uses the paper's symbols as identifiers.
What it provides
- The radiometric camera around any chief-ray model:
RadiometricCameraModel(the geometric camera, the focus surfaceFocus, the aperture radius and the vignetting — VisualRadiometryToolkitCore'sTensorProductVignettingandRadialVignetting, with the calibration's gauge pincenter_constraint) as one object, a photograph being the camera at a pose (render_view), and the radiometric back-projectionπ̂⁻¹. - The board: the printed board, VisualRadiometryToolkitCore's
TexturedTarget, with its own state (its warping, printed edge and in-plane print error:TargetState), its radiometryTargetRadiometry(the Cook–Torrance SVBSDFMaterialand the printed edge), and VisualRadiometryToolkitCore's spherical-harmonicsEnvironmentLight. - Image formation: the radiant flux of a pixel
Φ_pixeland the renderrender_view, by fixed quadrature rules on the pixel and the aperture and a Monte Carlo radiance with common random numbers. - Calibration: the
CalibrationState, the closed-form solves of the blocks the flux is linear in (solve_linear_blocks!), and the joint Levenberg–Marquardt with pathwise Jacobians (run_lm!,run_lm_grid!) on the CPU or a CUDA GPU, its residuals whitened when a RAW photograph's sensor model has a noise variance law (photo_set: VisualGeometryToolkitCore'sSensorModelresolved by the photograph's EXIF). - The pipeline: real sessions in, through a start (
start_calibration, its recordsave_calibration_start/load_calibration_start), the refinement (refine_calibration!) and its records out (save_radiometric_calibration) — orcalibratein one call — configured by one TOML file (PipelineOptions, defaultsconfigs/default.toml); the driver isbin/refine.jl; the API surface: API surface.
Its design is explained in The radiometric camera model: the model in the paper's symbols, the identifier ↔ symbol table, and the data conventions (COLMAP pixels, metres, the camera frame).
Installation
RadiometricCamera and the Visual Radiometry Toolkit packages it builds on (VisualRadiometryToolkitCore, VisualRadiometryToolkitIO) are registered in CauRegistry; the Visual Geometry Toolkit packages are in General. Add that registry once, beside General, then the package and the two stack packages the example uses (registration in General is planned):
pkg> registry add General https://github.com/prittjam/CauRegistry.jl
pkg> add RadiometricCamera VisualGeometryToolkitCore VisualRadiometryToolkitCore(registry add names General too, so that a new Julia installation gets both registries.)
Julia 1.13 or later. A CUDA GPU is optional: the CPU backend runs everything, more slowly.
A first example
A 320 × 240 px fisheye camera 1.5 m in front of a printed checkerboard: the example renders the radiant flux of every pixel the board covers, through the camera's aperture and focus surface.
using RadiometricCamera, VisualGeometryToolkitCore, VisualRadiometryToolkitCore
using RadiometricCamera: RenderSettings, compute_mask, render_view, sphere_radius, Φ_pixel, 𝔟
using VisualRadiometryToolkitCore: Material
# A printed 2 m × 2 m checkerboard of 8 × 8 squares, and a 320 × 240 px fisheye
# camera (Core's double-sphere lens) 1.5 m in front of it, looking at it.
texture = [Float64(isodd(fld(i - 1, 32) + fld(j - 1, 32))) for i in 1:256, j in 1:256, c in 1:3]
board = TexturedTarget([-1.0, -1.0, 0.0], [2.0, 0.0, 0.0], [0.0, 2.0, 0.0], texture; facing = [0.0, 0.0, 1.5])
W, H = 320, 240 # the image, in pixels
K = CameraCalibrationMat(412.0, 0.0, 0.0, 0.0, 412.0, 0.0, W / 2, H / 2, 1.0) # focal length 412 px, principal point at the centre
geometric = BoundedCamera(CameraModel(K; lens = Lens(DoubleSphere(0.25, 0.4))), Size2(W, H))
# The radiometric camera, one object, composed from the geometric camera: focused at 1.5 m, aperture radius 4 cm, and no
# vignetting (VisualRadiometryToolkitCore's NoVignetting, the default: V ≡ 1, no parameters).
camera = RadiometricCameraModel(geometric; focus = 1.5, aperture_radius = 0.04)
# The photograph's pose, camera → world, metres: lookat_pose(camera centre, the point it looks at, the world's up direction, which fixes the roll)
pose = lookat_pose(Vec3(0.05, -0.05, 1.5), Vec3(0.0, 0.0, 0.0), Vec3(0.0, 1.0, 0.0))
light = EnvironmentLight(SphericalHarmonicMap(1.0, 2), 𝔟(board), sphere_radius(𝔟(board), [pose.t])) # VisualRadiometryToolkitCore's light, L ≡ 1
settings = RenderSettings(; n_𝐩 = 2, n_ρ = 3, n_φ = 8, N = 4)
mask = compute_mask(camera, pose, board, settings; δ = 0.02, backend = RadiometricCamera.rc_backend("cpu")) # the device: rc_backend of a compute tag (public, not exported)
img = render_view(1, camera, pose, Material(), light, board, settings, mask) # photograph 1 (its index keys its Monte Carlo samples)
count(mask) # 76800 = W × H: the board covers every pixel
Φ = Φ_pixel(160, 120, 1, camera, pose, Material(), light, board, settings) # the pixel 𝐮 = (u, v) = (160, 120)
Φ ≈ img[120, 160, :] # true: the render is Φ_c(𝐮) of each pixel (img is H × W × 3: row v, column u)Documentation
Start here
- Getting Started: two rendered photographs, and the lighting, vignetting and material recovered from them.
How to
- Calibrate a camera: plug in a geometric model, calibrate a real session, run the driver
bin/refine.jl. - Set up the data and the hardware: the clone, what runs where (CPU, GPU), the published data.
- Run the tests and check a change: the suite's runs and the four checks of a change.
Explanation
- The radiometric camera model: the model, the identifier ↔ paper-symbol table, the code's layout and the data conventions.
Reference
Role in the stack
RadiometricCamera builds on the VisualGeometryToolkit stack: VisualGeometryToolkitCore's cameras, lenses, spline cameras (SplineLens) and thin-plate splines, VisualGeometryToolkitIO's records, VisualGeometryToolkitEstimation's fits and VisualGeometryToolkitFeatures' image readers. It builds on the Visual Radiometry Toolkit for the generic radiometric layer: VisualRadiometryToolkitCore's radiometric camera, focus surface, quadrature, material, textured target, vignetting models and light, and VisualRadiometryToolkitIO's records of the vignetting and the light. The experiments that reproduce the paper are experiments/ beside the package in its repository: staged drivers with configs of record, their inputs the published data tiers.
Citing
T. Harder, F. Seegräber, D. Nakath, H. Sauer, K. Köser, J. Pritts, Real Cameras are Radiometric, arXiv preprint (identifier to follow), and the code: https://github.com/RealCamerasAreRadiometric/realcamerasareradiometric.github.io.
This manual is published on the paper's project page, beside the paper and its supplement (PDF).