Putting it more philosophically, take 1+1 = 2. This does not mean: "Give me another and we'll have twice as much." The "ones" here live in the imaginary world of mathematics. They are the same as each other and do not refer to anything. In the real world, though, there are no ones or twos, just things.That's because math itself isn't just numbers, and the physical world itself isn't mathematical.
Modern physics is called "mathematical" because it makes use, in a remarkable way, of a quite specific kind of mathematics. But it is only able to proceed mathematically because, in a deeper sense, it is already mathematical. Τα μαθήματα means, in Greek, that which, in his observation of beings and interaction with things, man knows in advance: the corporeality of bodies, the vegetable character of plants, the animality of animals, the humanness of human beings. Along with these, belonging to the already-known, i.e., "mathematical," are the numbers. When we discover three apples on the table we recognize that there are three of them. But the number is something "mathematical." Only because numbers represent, so to speak, the most striking of the always-already-known, and therefore the best-known instances of the mathematical, is "the mathematical" directly reserved as a name for the numerical. The essence of the mathematical, however, is in no way defined in terms of the numerical. Physics is, in general, knowledge of nature. In particular, it is knowledge of material corporeality in motion; for corporeality manifests itself immediately and universally -- albeit in different ways -- in all natural things. When, therefore, physics assumes an explicitly "mathematical" form, what this means is the following: that through and for it, in an emphatic way, something is specified in advance as that which is already known. This specification concerns nothing less than what, for the sought-after knowledge of nature, is henceforth to count as "nature": the closed system of spatio-temporally related units of mass. Pertaining to this ground-plan, in accordance with its prior specification, are to be found, among others, the following definitions. Motion is change of place. No motion or direction of motion takes precedence over any other. Every point is equal to every other. No point in time has precedence over any other. Every force is defined as -- is, that is, nothing but -- its consequences as motion within the unity of time; and that means, again, change of place. Every natural event must be viewed in such a way that it fits into this ground-plan of nature. Only within the perspective of this ground-plan does a natural event becomes visible as such. The ground-plan of nature is secured in place in that physical research, in each step of investigation, is obligated to it in advance. This obligation, the rigor of research, has, at a given time, its own character in keeping with the ground-plan. The rigor of mathematical science is exactitude. Every event, if it enters at all into representation as a natural event, is determined, in advance, as a magnitude of spatio-temporal motion. Such determination is achieved by numbers and calculation. Mathematical research into nature is not, however, exact because it calculates precisely; rather, it must calculate precisely because the way it is bound to its domain of objects has the character of exactness.
P. 59-60