Temporal and kinematic properties of motor behavior reflected in mentally simulated action
Parsons, L. M. (1994). Temporal and kinematic properties of motor behavior reflected in mentally simulated action. Journal Of Experimental Psychology: Human Perception and Performance, 20(4), 709–730. doi:10.1037/0096-1523.20.4.709
Summary
In this paper, Parsons asks the question: how similar are mental simulations of motor actions to actual motor actions themselves? In a series of five experiments, he demonstrates that they correspond quite closely—though they differ in a few revealing ways—supporting the hyothesis that mental simulations do operate on a detailed model of the body and utilize the same type of trajectory optimization/motion planning that the motor system does.
Methods
In all experiments, Parsons showed participants images of left and right hands from six cardinal perspectives and 12 orientations ($30^\circ$, $60^\circ$, $90^\circ$, $120^\circ$, and $150^\circ$; clockwise and counterclockwise).
In Experiment 1, participants first performed a task in which they had to move their hand into the specified target position. They then performed another task in which they had to determine whether the specified target position depicted a right hand or a left hand (which was hypothesized to elicit mental simulation). Results showed that response times for the movement task and the left-right judgments were nearly identical, and that they increased as the distance between the original hand position and the target posture increased (in trajectory space). The cases where the RTs were less similar between the two tasks was for awkward and uncommon hand positions, in which case the response times for the left-right judgments tended to be longer (perhaps because people have a less good kinematic model of their hands in those positions).
Experiment 2 was the same as Experiment 1, except that instead of making left-right judgments, participants were told to imagine moving their hand to the target position, thus explicitly engaging them in mental simulation. The results were pretty much the same as in Experiment 1, suggesting that participants in the left-right judgments were indeed using mental simulation.
Experiment 3 was a control condition in which Parsons controlled for the time to perceive the target. Under this control, participants had RTs that were about 300ms faster on average, and which were especially faster for positions which were more awkward, suggesting that the awkward positions also take more time to perceive.
Experiments 4 and 5 had participants make left-right judgments (Experiment 4) or move to the specified target position (Experiment 5) but additionally had participants begin with their hands in different initial positions. The idea was to test whether mental simulation began from a canonical body pose, or from the current body pose. Parsons found that the original position did influence RTs in a way that was similar both for the left-right judgments and for movements.
Algorithm
n/a
Takeaways
This is such a cool set of experiments. What’s really remarkable about this is that participants can look at a schematized drawing of a hand, map it to their own hand (without looking at their own hand), and make the desired movement. Given the results that people take the same amount of time to make the right-left judgments, or just to explicitly perform the mental simulation, this is fairly strong evidence that people have a detailed mental model of their own body that they can use for motion planning and trajectory optimization.
One might say that it is not necessarily the case that this mental model need be kinematic—it could just be that people are determining the pose that they need to reach in configuration space, that they know their current pose in configuration space, and that then they compute some trajectory optimization and execute the resulting plan. But, it wouldn’t really make sense why people would need to do this for left-right judgments: if all it were was that people were trying to reach some point in configuration space, then they would presumably have the information already as to which hand it was. I suppose, though, that even if they have a kinematic model they wouldn’t necessarily need to do a mental simulation in the left-right task if they’ve already mapped the stimulus onto their body. Or, perhaps, mapping the stimulus onto the body is the mental simulation? It might be revealing to, instead of controlling for the perception time (as in Experiment 3), have people look at the stimulus and then indicate when they are ready to make the motion. If the perception itself is what is making use of the mental simulation, then that RT should be the same as the RT in the left-right judgments. But that would still just make things more puzzling: how is it that the mental simulation seems to match the actual movement if the point of the mental simulation is to determine how to make the movement?
To be more precise, in robotics, it seems that the process is something like:
- Determine initial and goal states in configuration space.
- Perform trajectory optimization to find a path from the start to the goal.
- Execute the found trajectory.
So, one could say that a method of solving the right-left judgment would just be the first step of the above process: if you know the goal state, there is no need to actually compute and execute a trajectory. Perhaps the reason why we see something that looks like computation and execution of the trajectory is that we don’t deterministically compute the goal state. Instead, perhaps we form hypotheses about what the goal state is, and then attempt to compute multiple trajectories under these different hypotheses until we find one. If computing the trajectory is linear in the distance of the trajectory, then this should reflect the observed behavior.
A further question I thought of relates our mental simulation of action to the mental simulation of the motion of objects. Trajectory optimization in robotics tends to usually be in terms of control (i.e., torques); if the same is true in humans, and our mental simulations reflect this, then does that extend to our mental simulation of other objects? That is, do we mentally simulate the control of those objects (e.g. in mental simulation do we imagine that they are being rotated by some entity) or just the kinematic motion of those objects? Presumably, it is at least sometimes the control (e.g. if you imagine grasping a coffee cup and taking a sip of coffee), but presumably at other times it is not (e.g. if you imagine a leaf falling from a tree). In the mental rotation case, my intuition is that it really could be either one. Understanding this distinction is going to be an important question, I think.