Robust Mesh Watermarking Emil Praun Hugues Hoppe Adam Finkelstein Princeton University Microsoft Research This talk presents a method for inserting watermarks into 3D models in such a way that they are difficult to remove. This was work done in collaboration with Hugues Hoppe from Microsoft Research and with my advisor at Princeton, Adam Finkelstein.

Watermarking Applications Authentication / localization of changes Fragile watermarks Ownership protection Robust watermarks Tracing of distribution channels Fingerprints Watermarking is actually a broad term. It commonly refers to a variety of applications, which have little in common in terms of requirements and implementation. For authentication of content and localization of changes we need a watermark that is removed or affected by the slightest changes to the document. These schemes are called fragile watermarks. For ownership assertion we need a watermark that will resist any attempts at removing it, and that will also withstand format conversions and routine processing. Such watermarks are called robust. A particular type of robust watermarks are called fingerprints. By watermarking each sold copy of the document with a distinct watermark, a vendor would later be able to identify the initial buyer of a pirated copy.

Watermarking Applications Authentication / localization of changes Fragile watermarks Ownership protection Robust watermarks Tracing of distribution channels Fingerprints Our work pertains to robust watermarks, used for ownership protection. For the remainder of the talk, I am going to focus on these.

Motivating Scenario 1. Alice creates a 3D shape, and publishes it on the web. 2. Bob sells it as his own. Here is the motivating scenario: The good character, Alice, creates a 3D model and publishes it on the Web. The bad guy Bob steals the model, maybe modifies it a little bit, and then starts selling it as if it were his own. Alice notices Bob selling her model, and wants to sue him. But how can she prove that she is really the owner? 3. How can Alice prove ownership? (and make Bob pay her a lot of money)

Digital Watermarks ? original document watermark Hidden in data! kept secret insertion Hidden in data! published original document watermark Here is how a digital watermark might help: We start with the original document and a secret watermark, which is just a vector of m random numbers. Through watermark insertion, we hide the random numbers in the document’s data. We keep secret the original and the watermark and we publish the watermarked document. Someone takes it, modifies it, or “attacks” it, and we recover it as the suspect document. Through watermark extraction, we get a new vector of numbers, Which we compare against the one we inserted. If they are very similar, we know that the suspect document came from the one we watermarked. ? suspect document “attack” extraction detected watermark

Incidental Attacks Filtering & smoothing A/D & D/A conversions Scaling Rotation Cropping A robust watermark should survive a wide variety of attacks, both incidental – meaning modifications applied with a purpose other than to destroy the watermark – and malicious. Examples of incidental attacks might be filtering and smoothing, analog to digital and digital to analog conversions, translation, rotation, scaling, cropping ..

Malicious Attacks Adding noise Adding another watermark Resampling Statistical analysis Here are some examples of malicious attacks, or attacks designed specifically to remove or weaken the watermark: Addition of noise to the data samples, Insertion of multiple watermarks in the same document, Resampling – in the case of 3d models this might include mesh simplification Or the use of some statistical analysis tools on the watermarked document for the purpose of determining information on the secret watermarked vector, of the places where this was hidden. Armed with such information an attacker might remove the watermark.

Our Goal Watermarking scheme for 3D models: Robust against attacks Works on arbitrary meshes Preserves original connectivity Imperceptible

Previous Watermarking [Cox et al. ’97] Introduce spread-spectrum for images [Ohbuchi et al. ’98] 3 schemes fragile under resampling [Kanai et al. ’98] Requires subdivision connectivity meshes [Benedens ’99] Redistributes face normals by moving vertices There has been a lot of work in watermarking sound, images and video. Most current successful approaches are based on spread-spectrum principles, introduced by Cox et al. from NEC in a paper published in 97. I am going to come back to explain these principles. 3D watermarking has received attention only recently. In 98 Ohbuchi et al have proposed 3 schemes that work by watermarking ratios of edge lengths, ratios of tetrahedra volumes, or by changing the connectivity of the meshes to embed visual patterns in the tessellation. All these schemes are fragile under resampling. Kanai et al. proposed watermarking meshes by changing the wavelet coefficients of the multiresolution representation. The downside of the method is that it requires subdivision connectivity meshes. In January of this year Benedens introduced a robust scheme that works by perturbing vertex positions in an optimization loop to redistribute face normals.

Spread-Spectrum Watermarking Transform to frequency space [Cox et al. ’97] DCT Our method draws its robustness from the spread-spectrum principles. Cox et al. introduced these principles for image watermarking: they transform the image to frequency domain, using a discrete cosine transform, then modify a large set of the frequency coefficients, and then transform back. The key ideas are to modify many coefficients, image frequency domain

Salient features  largest coefficients Spread-Spectrum Salient features  largest coefficients Perturb coefficients slightly to embed signal and also, to affect the most important coefficients, the ones with the largest absolute values. The reason behind this is that if the watermark is hidden in features of low significance, an attack might alter those features, and remove the watermark, without degrading the document. If we want the watermark to be robust, we must hide it in the most important areas of the document. Of course, there is a tradeoff: if the watermark is too strong, we can easily degrade the document when watermarking it. The scheme works by adding small random numbers to certain DCT coefficients. This is equivalent to adding a foam-mattress – looking basis function to whole the image. Image basis function  DCT coefficient

Our Approach Extend spread-spectrum method to meshes Problem: no DCT Solution: multiresolution representation Problem: no natural sampling Solution: registration & resampling Our approach consists of extending the spread-spectrum method to meshes. There are a couple of problems that become evident when one tries to do so: Since meshes can have arbitrary topology, there is no frequency transform. We address this problem by extracting frequency information from a multiresolution representation of the mesh. Second, in the case of meshes, we don’t have the nice uniform grid sampling used for images, so in the case when the attack changed the sampling of the data, such as in a simplification attack, we must revert to the original sampling in order to extract the watermark. A coordinate frame registration might also be required for attacks that involve translation, scaling or rotations.

Replacing DCT Basis Functions image mesh ? cosine basis Addressing the first problem, recall that for images, adding a small number to one DCT coefficient is equivalent to adding a cosine tensor product basis function to the image. We need something similar for arbitrary surfaces. Our basis functions will have finite support, and we obtain frequency information from a multiresolution analysis of the mesh. Specifically, we have used a Progressive Mesh representation, based on restricted edge collapses. Progressive Meshes were introduced by Hugues Hoppe at Siggraph in 96. Multiresolution  frequency information Progressive mesh [Hoppe ’96]

Multiresolution Neighborhoods vertex neighborhood corresponding mesh region We start with a fine representation of the model, which we simplify using a sequence of edge collapses. During one of these operations, an edge is replaced by one of its endpoints. Here, the blue edge is collapsed to the red vertex. The reverse operation is vertex split, or vertex add: an old vertex gives birth to a new vertex and edge. We select the most important vertex splits, the ones for which the location of the new vertex is hardest to predict from the existing neighbors. We then track these neighborhoods through the sequence of refinement operations up to the original fine mesh. These fine neighborhoods define the support of the basis functions. Since we selected the vertex splits that carry the most information, their neighborhoods correspond to important features. The difference between the actual and predicted positions of the new vertex provides hints on the allowable perturbation on the region. Naturally correspond to important features Provide hints on allowable perturbation

Scalar Basis Function i amplitude  i direction di Each basis function is characterized by a scalar value at each vertex in the support and by a global displacement direction, given by the normal at the center. To obtain the scalars, we compute a relative radius function, varying uniformly from 0 at the center vertex to 1 on the boundary. We then map this radius through a function of our choice. We have experimented with several mapping functions and we have settled for what we call the ‘sombrero’. This function was chosen to have C1 continuity at the apex and at the boundary, so that one would not notice the watermark. It also had the constraint that the integral over the disc be zero so it would be volume preserving when applied to a circular region. radius displacement

Watermark Insertion Construct basis functions  1 …  m To insert the watermark, we first construct the basis functions, and then perturb the mesh vertices. For a watermark of m coefficients, we use m different basis functions which may overlap. In the right image, the black outline shows the boundary of the dark blue basis function, which is partially overlapped by the brown, yellow and magenta ones.

Watermark Insertion Construct basis functions  1 …  m Perturb each vertex: basis function coefficient watermark direction The displacement for one vertex is given by the sum of the influences of all basis functions whose support contain the vertex. Each basis function is weighted by a coefficient w from the secret watermark vector. These coefficients are generated randomly using a normal distribution. Given the constructed basis functions, which do not depend on the random vector, but only on the initial mesh, we can express watermark insertion in matrix form: the new vertex positions v’ take the value of the old positions v plus the watermark vector multiplied by the matrix B. watermark coefficient Matrix system:

Watermark Extraction Get points v* on attacked mesh surface corresponding to original mesh vertices v Use same basis functions  1 …  m and hence same matrix B Solve least-squares system for w*: To extract the watermark, we first bring the suspect mesh into the same coordinate frame as the original and we then pair vertices of the original mesh with points on the surface of the suspect mesh. We’ll come back to this in a minute. Using the same basis function matrix B we solve a linear least squares system to determine the coefficients w* of the suspect watermark. The right hand side of the system is given by the motion residuals of the original vertices.

False-Positive Probability Correlation  = < w*,w > Pfp computed from  and m using Student’s t-test Declare watermark present if Pfp < Pthresh ( e.g. Pthresh = 10-6 ) Finally, we compare the extracted watermark w* with our secret watermark w and obtain a probabilistic answer: the false-positive probability tells us how likely it is that the suspect model was not derived from our watermarked model. The lower this probability is, the more confident we are that the model was stolen. The probability computation takes into account the linear correlation between the 2 watermark vectors and also the number of recovered coefficients, m, on which we based the computation. If a binary yes/no answer is required, we compare the computed probability with a threshold chosen a-priori based on the legal requirements of the application. For example, we might declare the watermark present if the probability that the two vectors were this similar by accident is less than one in a million.

Process (1) original mesh (2) watermarked (exaggerated) Here is a recap of the whole watermarking process: we start with the original mesh and a vector of random numbers, which we keep secret. We watermark the mesh and publish it. The watermarked model is perceptually identical to the original. In the upper right corner you see how the model would look like if the watermark strength would be exaggerated ten times. In figure 3 you see the suspect mesh – the fandisk was simplified, rotated and scaled. Once we have the suspect mesh, before extracting the watermark coefficients from the vertex residuals, we might need to register and resample the mesh. (3) suspect mesh (4) registered (5) resampled

Registration & Resampling [Chen & Medioni ’92] Resampling choices: Closest point projection Ray-casting along local normal Global deformation of original For registration we use a standard technique developed by Chen and Medioni in 92. It is relatively fast and can work with partial - or cropped - models. For the right hand side of the least squares system, we need to have a corresponding point on the suspect mesh for each of the original vertices. So we would like to resample the suspect mesh to the original connectivity. Several choices present themselves: we can project points on the suspect surface, or we can try to cast a ray in the direction of the local normal and intersect the other surface. It turns out both these approaches have serious problems, especially near sharp corners, so we have opted for a global deformation of the original mesh towards the suspect mesh geometry.

Global Deformation Deform original mesh to fit suspect mesh Minimize: Inter-mesh distance ( vertex springs ) Deformation ( edge springs ) Penalty for flipped triangles Accurate, but slow Suspect mesh To perform the deformation, we optimize on the coordinates of the mesh vertices, while minimizing an energy functional. This functional tries to bring the mesh close to the suspect mesh, while keeping the “aspect” of the original mesh. To do this, the functional simulates springs placed on the edges. The spring rest length is the original length of the edge. Since in some areas, due to the watermark or to the attack, the surface might have shrunk, trying to keep the edges to the original length might cause the surface to buckle over. We really don’t want that, so we introduced a term that severely penalizes these cases. This approach yields goods results, even if the points we obtain are not on the suspect mesh, but close to it. One drawback though is that the process is slow. Optimized mesh

Results watermarked mesh 1/2 faces similarity watermarked mesh noise 10-7 10-29 watermarked mesh 1/2 faces similarity 10-6 10-7 Here are some results from test cases we ran. In the left column you can see the watermarked meshes, and on the right are the results from various attacks. In the upper right corners you can see the false-positive probability with which the watermark was detected. Again, that number gives the chance that we are wrong when we say that the watermark is there. The lower it is, the happier we are. In the top row, the dragon was simplified to half the number of faces. For this test, the chance of this being a false positive is one in ten million. In the upper right corner, the attack consisted of translation, rotation and scaling. In the middle bottom picture you see the mannequin head with white noise added to the vertex coordinates. Even in the case of this severe attack, we are able to claim that the watermark is present with near certainty. watermarked mesh noise 2nd watermark

Results watermarked mesh 1/8 faces cropped watermarked mesh smoothing 10-13 watermarked mesh 1/8 faces cropped 10-12 10-2 Here are some more results. Note that in the case of the crop attack, we lose some watermark coefficients – the ones whose support was entirely on the tail of the bunny. Also, the basis functions corresponding to other coefficients have truncated support. However, we are still able to recover these coefficients exactly from the remaining vertices in the support, so the false-positive probability of guessing the original coefficients exactly is zero. In the bottom right you can see the results of a really extreme attack. We subjected the fandisk to addition of noise, smoothing, simplification, translation, rotation, scaling, simplification to half the number of faces, and cropping. Even then, the false positive probability is still of one in a hundred. watermarked mesh smoothing all attacks

Summary Robust watermarking for 3D meshes Spread-spectrum Basis functions from multiresolution analysis Resampling as global optimization Resilient to a variety of attacks In conclusion, we have implemented a robust watermarking scheme for 3d models. The robustness comes from the principles of spread-spectrum, which we have applied to surfaces. To replace the DCT analysis used for images, we have created a set of basis functions, using information from the multiresolution analysis of the mesh. For watermark extraction, we needed a resampling procedure, which we have implemented robustly as a global deformation. We have tested the method against a wide variety of attacks, and have obtained good results.

Future Work Consider other attacks: Explore other basis functions General affine and projective transforms Free-form deformations! [StirMark by Petitcolas] Explore other basis functions e.g. [Guskov et al. ’99] Fast mesh recognition  web crawler Some directions for future work might consist of improving the method against a larger class of attacks. It is fairly easy to adapt the registration ho handle affine of projective transforms. Free-form deformation, or lattice deformations, are more problematic however. We note that to date they are the most effective attack against image watermarking as well. Another possibility for future work might be to explore the use of other basis functions, for instance, the ones generated by the parameterization introduced by Guskov et al. It would also be very useful to have a fast mesh recognition scheme that would allow us to create automatic watermark verifiers for the web. Watermark extraction is expensive due to the resampling optimization. It would be useful to be able to cull out models of other objects than the one we are looking for.