Getting Started
This tutorial renders two photographs of a printed board through a radiometric camera whose lighting, vignetting and material are known, then starts a calibration from wrong values of those three blocks and recovers them from the photographs. Every block below runs when the documentation is built.
The scene and the camera
The board is a printed 2 m × 2 m checkerboard (a 256 × 256 texture of 8 × 8 squares, linear values in [0, 1]). The geometric camera geometric is VisualGeometryToolkitCore's pixel camera: a BoundedCamera of a CameraModel whose K is in the image's COLMAP pixels, over a double-sphere fisheye lens; any Core lens plugs in the same way. The image is W × H = 320 × 240 pixels (the BoundedCamera's Size2: the sensor the camera images onto), small enough that this page runs in seconds; a real calibration runs at the sensor's full resolution. The principal point is the image's centre and the focal length is 412 pixels.
using RadiometricCamera, VisualGeometryToolkitCore, VisualRadiometryToolkitCore
using VisualGeometryToolkitCore: cost # Core's public `cost` (not exported): ½ Σ‖Φ − ϕ‖² of the state, here
using VisualRadiometryToolkitCore: Focus, Material, with_vignetting # the generic radiometric types: the focus surface, the material, a camera's parts
using RadiometricCamera: RenderSettings, compute_mask, render_view, solve_linear_blocks!, sphere_radius, 𝔟
texture = [Float64(isodd(fld(i - 1, 32) + fld(j - 1, 32))) for i in 1:256, j in 1:256, c in 1:3]
board = TexturedTarget([-1.0, -1.0, 0.0], [2.0, 0.0, 0.0], [0.0, 2.0, 0.0], texture; facing = [0.0, 0.0, 1.5])
W, H = 320, 240 # the image, in pixels
K = CameraCalibrationMat(412.0, 0.0, 0.0, 0.0, 412.0, 0.0, W / 2, H / 2, 1.0)
geometric = BoundedCamera(CameraModel(K; lens = Lens(DoubleSphere(0.25, 0.4))), Size2(W, H))
settings = RenderSettings(; n_𝐩 = 2, n_ρ = 3, n_φ = 8, N = 4)RenderSettings fixes the quadrature: n_𝐩 × n_𝐩 nodes on each pixel, n_ρ × n_φ on the unit aperture disc, and N Monte Carlo samples of the radiance (common random numbers: the same samples at every evaluation, so the flux is a smooth, deterministic function of the parameters).
Two poses
The camera takes two photographs, from two poses (camera → world, metres) around the board.
# lookat_pose(camera centre, the point it looks at — the board's centre, the world's up direction, which fixes the roll)
poses = [lookat_pose(Vec3(0.3cos(k), 0.3sin(k), 1.4 + 0.1k), Vec3(0.0, 0.0, 0.0), Vec3(0.0, 1.0, 0.0)) for k in 1:2]
b = 𝔟(board)
r_SH = sphere_radius(b, [p.t for p in poses])The lighting lives on the sphere around the board's centre 𝔟 of radius r_SH, which sphere_radius chooses to hold every camera centre (the pipeline's sphere_radius(board, centres) also every corner of the page, so that no light ray from the board misses the sphere).
The truth, and its photographs
The true radiometric camera is one object, a RadiometricCameraModel: the geometric camera geometric, a focus surface at 1.5 m (Focus: a constant inverse focus distance on a 3 × 3 node grid of the sensor), an aperture radius of 4 cm (no image distance: the defocus is the focus surface's) and a vignetting that falls off quadratically from the sensor's centre. The true lighting is a coloured spherical-harmonics field of degree 2, and the true material has the diffuse weight μ = 0.4. Each photograph is the camera at one of the poses: compute_mask and render_view take the camera and the pose (and the photograph's index k, which keys its Monte Carlo samples).
xs = range(-0.5, 0.5; length = 4)
vignetting_true = VisualRadiometryToolkitCore.with_control_points(TensorProductVignetting((4, 4), Rect(0, 0, W, H); degree = 2), [1 - 0.4 * (x^2 + y^2) for x in xs, y in xs])
camera_true = RadiometricCameraModel(geometric, Focus(3, 1.5, W, H), 0.04, vignetting_true) # geometric camera, focus surface, aperture radius (m), vignetting
a = zeros(3, 9); a[:, 1] .= (2.0, 1.9, 1.8); a[:, 3] .= (0.4, 0.3, 0.2)
light_true = EnvironmentLight(SphericalHarmonicMap(a, 2), b, r_SH) # VisualRadiometryToolkitCore's light: degree 2 per channel on the sphere (b, r_SH)
material_true = Material(; diffuse_weight = 0.4)
masks = [compute_mask(camera_true, p, board, settings; δ = 0.02, backend = RadiometricCamera.rc_backend("cpu")) for p in poses]
photos = [render_view(k, camera_true, poses[k], material_true, light_true, board, settings, masks[k]) for k in 1:2]
size(photos[1])(240, 320, 3)Calibrating the radiometry
A CalibrationState holds one camera, the poses of its photographs, the lighting, the material and the photographs ϕ. Here the geometry is known: the state starts from the true camera with a flat vignetting (with_vignetting), a grey lighting and the wrong weight μ = 0.7; Core's cost is ½ Σ‖Φ − ϕ‖² and the relative RMS residual.
camera_start = with_vignetting(camera_true, TensorProductVignetting((4, 4), Rect(0, 0, W, H); degree = 2)) # V ≡ 1
s = CalibrationState(board, settings, camera_start, poses, Material(; diffuse_weight = 0.7), EnvironmentLight(SphericalHarmonicMap(0.5, 2), b, r_SH), photos;
masks)
cost(s)(1545.8825050696837, 0.4925112421717906)The flux is linear in the lighting Θ_l with the vignetting held, linear in the vignetting Θ_v with the lighting held (bilinear in the two together) and affine in μ, so each block alone has a closed-form solve. Block by block, the solves creep along the coupling of Θ_l and Θ_v; solve_linear_blocks! with joint = true solves the three together instead (the pixel quadratures run once, into PixelParts; then solve_radiometry_gn!, a Levenberg–Marquardt on those parts) and reproduces the photographs exactly:
solve_linear_blocks!(s; joint = true)
cost(s)[2] < 1e-10, s.mat.diffuse_weight ≈ 0.4(true, true)The vignetting and the lighting come back up to their common gauge: the state pins V = 1 at the sensor's centre, and the lighting absorbs the true field's value there.
Next
The geometric blocks — the poses Θ_e, the lens Θ_d, the focus Θ_f and the aperture radius r — enter the flux nonlinearly. run_lm! refines them jointly with the radiometry by Levenberg–Marquardt with pathwise Jacobians, and run_lm_grid! does so for the generic grid model, tile by tile. Calibrate a camera does this on real sessions; The radiometric camera model states the model these blocks belong to.