# Physical Intelligence canonical_url: https://www.3.14financialcontents.com/en/capabilities/physical-intelligence/ markdown_url: https://www.3.14financialcontents.com/en/capabilities/physical-intelligence/index.md language: en content_type: capabilities status: published description: We work with computer vision, sensing, signal fusion, state estimation, simulation, navigation, motion planning, control, manipulation and edge computing. We explore how to integrate these capabilities with models, software and… Extending perception, decision and action from digital systems out into the physical world. ## What we build and research - observing through vision and sensors - estimating the state - representing the environment - planning - optimising a trajectory - controlling - manipulating - coordinating equipment - acting locally - verifying the effect - recovering or stopping the system when uncertainty exceeds what is permitted ## Technologies we build, apply and evaluate - OpenCV - AprilTag - vision models - calibration and pose estimation - sensor fusion - CUDA - TensorRT - Newton - NVIDIA Warp - OpenUSD - software in the loop and hardware in the loop - cuRobo - distance fields - collision checking - trajectory generation and optimisation - MPC and predictive control - geometric control - P/PD and identification - NVIDIA Jetson - Python and FastAPI - NATS/JetStream - durable workflows - MCP tools The system turns partial measurements into an estimate of the state with the confidence needed to decide. A physical operation changes the environment. To carry it out, the system relates partial measurements, an estimate of the state, objectives, constraints and mechanisms able to act. A robot moves, brings a tool close, picks up an object, modifies a position or coordinates its movement with other machines. The action takes place under geometry, time, friction, play, load, noise, latency and mechanical limits. Physical intelligence allows a system to relate perception and action within those conditions. This line extends operational intelligence to the physical world from the Laboratory. Information remains the basis. The system needs to know the objective, the state, the constraints and the authority. The difference is that now part of that information comes from incomplete measurements, and part of the result has to be checked against an environment that changes. ## Perceiving is estimating A camera delivers images. An IMU delivers accelerations and angular velocities. An encoder delivers positions or increments. The system turns those partial measurements into an estimate of the state with the confidence needed to decide. The estimate combines measurements, models and context in order to answer: - where the robot is; - what orientation it has; - what object it is observing; - with what confidence; - what movement has taken place; - what part of the environment remains uncertain; - which signal contradicts another; - what correction should be applied; - when the estimate no longer allows the work to continue. Treating perception as estimation makes it possible to design behaviours proportional to confidence. The system can carry on, reduce speed, look for a reference, change sensor, widen the observation, request intervention, return to a safe position or stop. Uncertainty is a variable that modifies the operation. ## Computer vision to understand geometry and state Vision can identify, locate, measure and track elements of the environment. We work with detection, segmentation, classification, pose estimation, markers and visual references, calibration, correspondences, depth, tracking, inspection, instrument reading and the relationship between image and geometric model. Usefulness depends on specific conditions: lighting, distance, movement, frequency, resolution, occlusion, optics, surface, latency and computing capacity. We evaluate vision within the physical journey: what decision changes, what tolerance it needs and what happens when the signal is missing or arrives late. A metric obtained over selected images describes those images; behaviour during an operation is measured in the operation. ## Signal fusion and state estimation Sensors bring different strengths. Odometry preserves continuity and accumulates error. A visual reference can correct drift and appear intermittently. An IMU contributes dynamics and also noise. An absolute encoder can offer a particularly valuable dimension of state. We fuse signals according to their frequency, uncertainty and geometric relationship. The architecture preserves timestamps, reference frames, calibrations, covariances or confidence measures, transformations, delays, validity, origin and sensor health status. In MontojOS, for example, the continuity of the movement can be sustained with odometry and IMU and corrected through visual references when they are available. The system preserves the state during the temporary absence of vision and knows how uncertainty grows while it lasts. ## Representing the environment and the body Planning needs a representation of geometry, joints, limits, tools, loads, permitted zones, obstacles, stations, tolerances, contacts, movement capabilities and safe states. The representation connects the digital model with the physical equipment. A trajectory that is valid in a model may need adjustments for flexion, play, wear, cabling, load, friction, manufacturing tolerance, calibration, inertia, deformation and unmodelled elements. That is why we distinguish nominal geometry, identified parameters and observed behaviour. The effect of a command depends on the state, the load, the geometry, the tolerances and the environment. ## Simulation to learn before the hardware Simulation makes it possible to explore behaviours, generate scenarios and evaluate decisions with greater speed and lower physical cost: validating geometry, detecting collisions, testing controllers, comparing sequences, generating data, studying sensitivity, training policies, reproducing failures, evaluating coordination and preparing test campaigns. Newton and NVIDIA Warp extend the ability to build GPU-accelerated simulations and to integrate physics, geometry and parallel computing within the Python ecosystem. Simulation provides a controlled environment. Hardware provides phenomena and tolerances that the model does not yet contain. The maturity journey connects the two: ```text MODEL ↓ SIMULATION ↓ SOFTWARE IN THE LOOP ↓ HARDWARE IN THE LOOP ↓ TESTS IN THE ENVIRONMENT ↓ BOUNDED OPERATION ``` Each stage answers different questions. The next step is enabled through explicit criteria. ## Navigation and mobility A mobile system needs to estimate its pose, represent routes and zones, plan, follow a trajectory, respond to obstacles, approach with precision, coordinate its movement with the rest of the operation and preserve the ability to stop and recover. We work with navigation that combines odometry, IMU, visual references, maps or known geometry, kinematic control, predictive control, terminal corrections, speed profiles and mission states. General navigation and the final approach may need different controllers. Cruising prioritises continuity and efficiency. The terminal phase prioritises lateral, longitudinal and angular error with respect to the task. The criterion of success consists of placing the system in a condition from which the next physical action can be executed within the necessary tolerance. ## Motion planning and optimisation Planning looks for a trajectory that meets objectives and constraints: position, orientation, collisions, joint limits, velocity, acceleration, jerk, time, energy, stability, load, forbidden zones and the ability to recover. cuRobo makes it possible to accelerate on GPU such tasks as inverse kinematics, collision checking, trajectory generation and motion planning. The acceleration widens the number of alternatives the system can examine within the time available. The final selection preserves real geometry, tools, payload, tolerances, controller limits, safety and verification. In MagdalenOS, planning can combine specialised backends with a representation of its own for the equipment and its constraints. ## Predictive control and adaptation A controller turns a reference into actions on the system. Predictive control uses a model to anticipate the evolution over a horizon and to choose actions that respect constraints. It can be useful in trajectory tracking, navigation, approach, stabilisation, manipulation, coordination and the management of limits. Its effectiveness depends on the fidelity of the model, the frequency, the latency and the ability to estimate the state. We work with geometric control, P and PD control and combinations, MPC, plant identification, constrained control, speed profiles, supervision, fallback and safe stop. Control is evaluated on error, stability, time, effort, recovery and behaviour under disturbances. ## Manipulation and mechanisms A manipulation operation connects geometry, position, orientation, contact, tool, load, sequence, actuators, sensors and verification. The system distinguishes approach, contact, grip, transport, release, withdrawal and safe position. An action may require coordination between base, arm, linear mechanism, magnets, shears, carriage, lifting or other actuators. The sequence preserves states and transition criteria. The acquisition or release of the object is confirmed through a signal of its own: encoder, current, position sensor, vision, change of load, actuator state or later observation. ## Edge: deciding close to the environment Part of the perception and of the control needs to run locally. The edge provides low latency, continuity during network losses, direct access to sensors, frequency control, privacy, the ability to stop and greater autonomy with respect to remote services. We work with platforms such as NVIDIA Jetson and other GPU or embedded architectures. The distribution can separate: **Local.** Critical perception, estimation, control, limits, stop, buffers and immediate recovery. **Remote.** Global planning, analysis, training, reporting, storage, supervision and high-level coordination. The boundaries of that division are designed from latency, connectivity, safety and cost of computation. ## Agents and physical operations Agents can help to interpret objectives, choose skills, coordinate missions, query knowledge, diagnose, prepare recovery, explain, ask for help and adapt a plan within limits. The critical physical trajectory remains protected by controllers, states and policies that can be verified. An agent can decide between declared actions; deterministic components maintain movement limits, permissions, interlocks, stop, critical sequences, contracts and completion criteria. This distribution makes it possible to take advantage of interpretation and planning while keeping the physical constraints in verifiable components. ## Coordination of equipment Physical Intelligence can extend from a single machine to a coordinated operation. The system represents equipment, capabilities, locations, tasks, resources, dependencies, zones, priorities, availability, failures, charge state and interfaces with people. Orchestration allocates work and preserves the global situation. Each piece of equipment keeps enough local control to operate safely. Coordination can combine workflows, events, optimisation, agents, planning, telemetry and human supervision. ## Verifying the action on the environment A physical operation ends when the environment makes it possible to observe its effect. Verification can answer: - did the robot reach the useful pose?; - is the object held?; - did the load change?; - did the station receive the item?; - did the mechanism return to a safe position?; - was the environment left ready for the next operation?; - can the machine carry on?; - has a condition been created that needs intervention? An independent signal increases confidence: the same chain that sends the command reports on its own execution, and checking the effect needs another observation. We design the criterion of success together with the task and incorporate it into the evaluation. ## Safety, stop and recovery Safety structures the physical operation. We design limits, zones, speeds, safe states, interlocks, supervision, stop, recovery, intervention, incident logging and testing. The ability to stop and return to a known condition is part of the result. Actions are classified by impact, reversibility, energy, proximity, confidence, authority and the ability to observe. An action with greater consequences demands more evidence, more control and more separation between planning and execution. ## What can change for an organisation ### Perception connected with decisions Vision and sensors stop producing detections alone and start modifying a defined operation. ### Greater local continuity The system can preserve perception and control through variations in connectivity. ### Better precision in bounded tasks Signal fusion and terminal correction help to reach tolerances that are useful for manipulation, inspection or interaction. ### Less hidden uncertainty The state and its confidence become observable and condition the behaviour. ### Faster planning GPU, simulation and optimisation make it possible to examine more alternatives within the time available. ### Checkable operations Each sequence incorporates a signal that makes it possible to distinguish command from effect. ### Safer learning Simulation, HIL and campaigns in the environment make it possible to discover limits before extending autonomy. ### Digital and physical integration Physical operations can be connected with ERP, workflows, events, people and reporting. ## Lines of work ### Visort Computer vision, geometry, detection, estimation and execution at the edge. ### MagdalenOS Equipment representation, planning, kinematics, manipulation, simulation and robotic coordination. ### MontojOS Navigation, signal fusion, terminal approach, physical mechanisms and verification of operations at defined stations. ## How we evaluate **Perception.** Precision, recall, geometric error, frequency, latency, calibration and behaviour by condition. **Estimation.** Position and orientation error, drift, stability, confidence and recovery after loss of signal. **Planning.** Solution rate, time, collisions, smoothness, margin, compliance with constraints and physical feasibility. **Control.** Error, stability, time, overshoot, effort, disturbances and recovery. **Operation.** Percentage of complete journeys, duration, retries, interventions, unsafe states and verified result. **Simulation.** Correspondence with hardware, coverage of scenarios and the ability to reproduce failures. **Safety.** Stop, limits, interlocks, recovery and response to invalid signals. **Maturity.** State by capability, equipment, task and environment. ## State of maturity Laboratory · capabilities under evaluation. Visort, MagdalenOS and MontojOS make it possible to integrate perception, estimation, planning, control and mechanisms within defined physical journeys. Each capability advances through simulation, software and hardware tests, campaigns in the environment and specific criteria for precision, safety, recovery and result. The website declares the state by task and environment. This way of presenting maturity makes it possible to widen the ambition while keeping each demonstration within its scope. ## Closing Physical Intelligence connects knowledge of the task with an estimate of the world and turns that estimate into actions under constraints. Its value appears when the system can perceive, decide, act, check and learn within the environment in which it will have to operate. --- https://www.3.14financialcontents.com/llms-en.txt · https://www.3.14financialcontents.com/llms-full-en.txt