Real-time 3D Eyelids Tracking from Semantic Edges
Real-time 3D Eyelids Tracking from Semantic Edges
QUAN WEN, FENG XU, MING LU, and JUN-HAI YONG, Tsinghua University
Fig. 1. We present an approach to reconstruct and track 3D eyelids in real time. This technique is integrated into a face and eyeball tracking system to obtain
full face results with more detailed eye regions, again in real time (blue rectangles show closeups of the eye region). Our technique successfully reconstructs
both eyelid shapes and poses, for instance the shapes of eye contours, double-folds and bulges in the center-left result and the pose differences between the
two eyes in the right-most result.
State-of-the-art real-time face tracking systems still lack the ability to realistically portray subtle details of various aspects of the face, particularly
the region surrounding the eyes. To improve this situation, we propose a
technique to reconstruct the 3D shape and motion of eyelids in real time. By
combining these results with the full facial expression and gaze direction,
our system generates complete face tracking sequences with more detailed
eye regions than existing solutions in real-time. To achieve this goal, we propose a generative eyelid model which decomposes eyelid variation into two
low-dimensional linear spaces which efficiently represent the shape and motion of eyelids. Then, we modify a holistically-nested DNN model to jointly
perform semantic eyelid edge detection and identification on images. Next,
we correspond vertices of the eyelid model to 2D image edges, and employ
polynomial curve fitting and a search scheme to handle incorrect and partial
edge detections. Finally, we use the correspondences in a 3D-to-2D edge
fitting scheme to reconstruct eyelid shape and pose. By integrating our fast
fitting method into a face tracking system, the estimated eyelid results are
seamlessly fused with the face and eyeball results in real time. Experiments
show that our technique applies to different human races, eyelid shapes, and
eyelid motions, and is robust to changes in head pose, expression and gaze
direction.
CCS Concepts: ? Computing methodologies ¡ú Motion capture;
Additional Key Words and Phrases: facial performance capture, eyelid modeling, eyelid tracking, semantic edge detection
This work was supported by the NSFC (No. 61671268, 61672307, 61727808) and the
National Key Technologies R&D Program of China (No. 2015BAF23B03). Quan Wen,
Feng Xu (corresponding author), Jun-Hai Yong are from TNList and School of Software.
Ming Lu is from TNList and Department of Electronic Engineering.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@.
? 2017 Association for Computing Machinery.
0730-0301/2017/11-ART193 $15.00
ACM Reference Format:
Quan Wen, Feng Xu, Ming Lu, and Jun-Hai Yong. 2017. Real-time 3D Eyelids Tracking from Semantic Edges. ACM Trans. Graph. 36, 6, Article 193
(November 2017), 11 pages.
1
INTRODUCTION
The human face is often the most important body part for a computer
to track as it conveys identity and emotion. Thus, facial capture and
animation is an important research topic in computer graphics, with
applications across movies, computer games, and online communications. Existing techniques reconstruct the face in real time using
consumer-level RGB or RGBD sensors, which makes obtaining facial
expression cheap and fast [Bouaziz et al. 2013; Cao et al. 2014a, 2013;
Li et al. 2013; Weise et al. 2011]. However, state-of-the-art methods
majorly focus on face skin regions and are still unable to realistically
convey many subtle expressions, such as the shape and motion of
the eyes¡ªthe window to the soul.
Many recent efforts have improved tracking for specific facial
organs, including eyes [B¨¦rard et al. 2016, 2014], eyelids [Bermano
et al. 2015], lips [Edwards et al. 2016; Garrido et al. 2016], and teeth
[Wu et al. 2016]. These techniques produce high-quality modeling,
but are too complex to be applied in real-time. Recently, real-time
eyeball modeling and tracking has been achieved [Thies et al. 2016b;
Wang et al. 2016; Wen et al. 2016] in face tracking and animation
systems. However, this alone is often insufficient for realistic capture.
Modeling eyelid shape and motion, including folds and bulges, is
still required to generate realistic eye regions in real time.
It is difficult to model the shape and motion of local eye regions
because they are small and their motions involve heavy occlusions
in fold regions. This is unlike capturing the full facial expression
where surfaces are larger and less often occluded. Recent works have
used principle component analysis (PCA) to model the overall shape
ACM Transactions on Graphics, Vol. 36, No. 6, Article 193. Publication date: November 2017.
193:2 ?
Quan Wen, Feng Xu, Ming Lu and Jun-Hai Yong
Fig. 2. Overview of our system. Note that the depth input is required by the face fitting technique, not our eyelid reconstruction. We show it here as it
contributes to the full face results shown in the figure.
of the eye region from 22 scans [Wood et al. 2016a,b]. However, eye
reconstruction should include subtle details like folds, eye contours,
and bulges, and real-time performance requires efficient estimation
of their motions. Shape from shading can be used to reconstruct
these details [Garrido et al. 2013; Richardson et al. 2016; Shi et al.
2014], but these systems are offline. In online systems, only midscale facial features, like strong wrinkles, can be reconstructed [Cao
et al. 2015].
In this paper, we propose a system to reconstruct more detailed
eye regions in real time. First, we propose a high-fidelity generative
eyelid model with a set of dimensions which independently represent the shapes and motions of eye contours, folds, and bulges. The
model is a linear parametric model which efficiently reconstructs
eye regions via linear combinations. Second, we propose projective
fitting of semantic eye region edges to reconstruct eye regions in
real time. The semantic eye region edges are extracted by a multichannel holistically-nested edge detector, which jointly achieves
edge detection and identification. Third, we propose a polynomial
curve fitting technique and a correspondence updating technique to
handle incorrect and missing edge results, and to achieve real-time
performance of the projective fitting.
We list our specific contributions:
? Real-time reconstruction, tracking, and animation of realistic
human eyelid shape and motion. It is combined with face and
eyeball tracking to generate more complete and vivid result.
? A linear parametric eyelid model for human eyelid shape and
motion. It represents details and fits real-time applications.We
will release the model to the community.
? A real-time semantic edge-based eyelid fitting solution. The
edge detector uses semantic information in a holisticallynested DNN model. Detailed eyelids are reconstructed in real
time by a novel 3D-to-2D projective edge fitting algorithm.
1.1
Related Work
We propose a more detailed eyelid model to represent eyelid shape
and motion, and use this model to achieve real-time eyelid reconstruction. As there is no previous work focusing on exactly the same
goal, we briefly survey models and reconstruction techniques for
general faces.
ACM Transactions on Graphics, Vol. 36, No. 6, Article 193. Publication date: November 2017.
Face Models. To better model, track, and animate human faces,
generative 3D face models have been proposed which involve prior
knowledge of face geometries and motions. Through these models,
a 3D face can be represented and reconstructed in a low dimensional
space. Blanz and Vetter [1999] propose the first 3D morphable face
model from 200 face scans of different individuals with fixed neutral
expression. The morphable face model is further extended with
1000 face scans [Booth et al. 2016] and with both shape and texture [Paysan et al. 2009; Zhu et al. 2015]. Besides facial identity,
facial expression is also represented in a low dimensional space.
The blendshape model, which consists of a set of key expressions of
a particular facial identity, can be used to represent novel expressions by linear combinations of blendshapes. Blendshapes can also
be generalized to novel identities by deformation transfer [Sumner and Popovi? 2004] or example-based rigging [Li et al. 2010].
The functions of morphable model and blendshape model can also
be achieved in one model, called a multilinear model [Cao et al.
2014b; Vlasic et al. 2005], in which facial identity and expression
are independently modeled by two sets of parameters.
These models focus on the overall face, but are not designed for
specific face parts. This is crucial to attaining higher fidelity in face
modeling. In [Edwards et al. 2016], a 3D viseme model, called Jali,
is proposed to model the speech-related mouth motion. Olszewski
et al. [2016] use a new blendshape model with 29 shapes for mouth
poses to better model the mouth regions (5 of them for modeling
tongue motions). Recently, teeth have been modeled delicately with
a tooth row model and a local shape model for individual teeth [Wu
et al. 2016]. For eyeballs, a morphable model has been proposed
to achieve lightweight eye capture [B¨¦rard et al. 2016]. For modeling eyelid and eye regions, early works focused on the domain
of 2D images, which are well surveyed [Ruhland et al. 2014]. In
recent years, 3D eye region models have been proposed [Wood et al.
2016a,b], where a PCA model with 8 dimensions is derived to model
the overall shape of eye regions in a fully opened eye pose. However,
there is no 3D generative model that can represent eye shapes with
folds and bulges, let alone model the eyelid dynamics at the same
time.
Face Reconstruction. We can generate high quality face shapes
and dynamics with stereo [Beeler et al. 2011] and shape from shading (SfS) techniques [Garrido et al. 2013; Shi et al. 2014]. For eye
regions, delicate eyeball models are reconstructed by a complex
Real-time 3D Eyelids Tracking from Semantic Edges ? 193:3
Fig. 3. Folds and bulges with different poses. The bottom image is zoomed
in from the left side eye region. (a,b) shows that the fold disappears with
the eye-closing motion. (c,d) shows that the bulge disappears with the
upwards-looking motion.
capture and processing technique [B¨¦rard et al. 2014]. An eyelid
morphable model is also proposed and used for image-based eye
region animation [Wood et al. 2017]. Neog et al. [2016] use a cubic
spline curve to build an eyelid shape model to learn the correlation
between eyeball direction and eyelid motion. Besides these largescale eyelid modeling and animation techniques, detailed dynamics
of eyelid folds are also modeled and reconstructed with multi-view
input and offline processing [Bermano et al. 2015]. None of these
techniques are real time, which limits their application.
Real-time 3D face tracking and animation was first demonstrated
by Weise et al. [2011] for a specific user with a RGBD sensor. This
work was extended by Li et al. [2013] and Bouaziz et al. [2013] to
remove the specific user requirement, and by Cao et al. [2013] to
require only RGB input. Later, Cao et al. [2014a] combined these
two extensions and reconstructed medium-scale face features like
strong wrinkles [Cao et al. 2015]. Occlusion handling can improve
the robustness of facial tracking systems [Hsieh et al. 2015; Liu
et al. 2015; Saito et al. 2016]. To improve realism, attention has
now turned to face parts. Wang et al. [2016] and Wen et al. [2016]
track eyeball rotations from RGB and RGBD input in real time.
In addition, following the works of real-time face reenactment on
RGBD and RGB inputs [Thies et al. 2015, 2016a], Thies et al. [2016b]
achieve gaze retargeting for face reenactment with virtual reality
(VR) headsets. This is similar to other research on VR-based face
tracking [Li et al. 2015; Olszewski et al. 2016]. All the aforementioned
real-time systems handle neither precise eyelid shapes nor motions.
1.2
Overview
Our system (fig. 2) first proposes two linear models for representing
the eyelid shape and pose in low dimensional subspaces (section 2).
Meanwhile, we train an improved holistically-nested network to
jointly achieve detection and identification of four semantic edges
in the eye regions (section 3). In the online tracking stage, with
the input color and depth sequence, we integrate our eyelid fitting
technique into a face and eyeball tracking system introduced by
Wen et al. [2016]. As the eyelid models have been pre-aligned to the
face models, we directly perform eyelid shape and pose tracking
(section 4) in the face tracking system. By iteratively solving the
optimizations, we recover the eyelid model parameters that best fit
Fig. 4. Example bases in shape and pose rigs. (a) the basic eyelid mesh: b0id
exp
and b0 ; (b-d) three shape bases with eye contour change, fold change
id , b id and b id ; (e-g) three pose
and bulge change on the right-side eye, b11
21
23
bases with downward motions on the inner part, the outer part and the
exp
exp
exp
whole of the upper eyelid of the right-side eye, b3 , b5
and b1 .
the four edges detected on the image. Finally, as our eyelid models
are generative, we can transfer the reconstructed eyelid motions to
a novel eyelid identity in real time.
2
LINEAR EYELID MODELS
For 3D face modeling, previous works use morphable models to represent the identity/shape changes of a face, and blendshape models
to represent expression/pose changes. Similarly, we also use two
linear models, with two sets of 3D mesh bases, to represent the
shape and pose variations of eyelids independently. In this manner,
users can control the shape and pose more freely. However, for faces,
the shape and pose correspond to totally different changes on face
geometry, while for eyelids, they may cause the same changes. For
example, different identities may or may not have folds and bulges,
and different poses (e.g., opened/closed, looking forwards/upwards)
may also cause the folds and bulges to appear and disappear, as
shown in fig. 3. Our two linear rigs share identical bases to handle
this issue.
2.1
Shape Linear Rig
Eyelids vary among genders, races, and ages. The characteristics of
an eyelid can be categorized as follows:
? Position stands for the relative positions of the eyes on a face.
It is represented by the vertical location of the eye pair and
the horizontal distance between the two eyes.
? Contour shape of the eye is affected by whether the eyes are
round or flat, wide or narrow in the horizontal direction,
upturned (the outer eye corner is higher than the inner eye
corner) or downturned (the outer eye corner is lower than
the inner eye corner).
? Double-fold is described by the distances between the fold and
the upper eyelid on different parts, and the strength of the
fold. Note that a single-fold eyelid, commonly seen in Asian
populations, is an extreme case of the double-fold eyelid. It
can be represented as a zero-strength double-fold eyelid.
? Bulge indicates the shape and strength of a bulge beneath an
eye. ¡®No bulge¡¯ is represented by a zero-strength bulge.
Based on the observations above, we asked an artist to build a set
of eyelids to cover the characteristic variations. We asked the artist
ACM Transactions on Graphics, Vol. 36, No. 6, Article 193. Publication date: November 2017.
193:4 ?
Quan Wen, Feng Xu, Ming Lu and Jun-Hai Yong
Fig. 5. Detection and identification of the four semantic edges. Each color
corresponds to one kind of semantic edge (blue: fold edge; green: top edge;
red: bottom edge; purple: bulge edge). (a,b) examples with all four edges; (c)
an example without fold edge; (d) an example without bulge edge.
to first build a basic 3D eyelid mesh with a neutral identity in a neutral, open pose, and then modify the basic mesh to generate a set of
new meshes. The artist edited these meshes to cover the eyelid space
as much as possible, based on her experience and photographic reference: . The difference between each new
mesh and the base mesh represents one dimension of one characteristic, with the topology and the semantic meaning of the vertices
kept between all meshes. We use this set of eyelids to construct the
bases of our linear shape model:
B id = {bkid |k = 0, ..., N id ? 1}, N id = 29,
(1)
b0id
where
is a pair of neutral eyelids. Details of the bases in the
shape rig, along with the following pose rig, are described in the
appendix, and we show a few eyelid bases in fig. 4. Note that for
position, the two eyes always move together (upper or lower, closer
or farther), but for other characteristics, the two eyes could change
independently as they may be slightly different for most people.
Thus, our bases either relate to two eyes or to one eye.
With the shape bases and a set of blending weights, the eyelid
model of a specific user in a neutral, open pose is synthesized by:
E N = b0id +
where w id = (w 0id , ..., w idid
N
2.2
id ?1
N?
k =1
?1
w kid (bkid ? b0id ),
(2)
)T is the shape weight.
Pose Linear Rig
We build the pose rig by manually generating a set of bases to cover
all possible geometric changes caused by eyelid poses:
exp
B exp = {bk
|k = 0, ..., N exp ? 1}, N exp = 23,
(3)
exp
where b0 is the same as b0id . Note that eyelid fold and bulge may
change with either shape or pose. Thus, there are some shared bases
for both shape and pose rigs to control the eyelid fold and bulge.
In practice, we first use the shape rig to estimate E N for a particexp
ular user. However, since b0 is different from E N , B exp cannot
be directly used to construct the pose rig for E N . To overcome this
drawback, we use deformation transfer [Sumner and Popovi? 2004]
¡ä
to recover B exp by transferring the deformation gradient between
ACM Transactions on Graphics, Vol. 36, No. 6, Article 193. Publication date: November 2017.
Fig. 6. Comparisons of different solutions on edge detection and identification. (a) input images; (b) results of four separate HEDs; (c) results of our
network with separate loss defined in eq. (8); (d) results of our network with
uniform loss defined in eq. (6); (e) ground truth. Note that as both (b) and
(c) separately consider the four edges, they always detect all the four edges
no matter whether or not the images contain fold edges or bulge edges.
exp
b0
exp
and bi
to E N . Correspondingly, the pose rig is used as:
exp ¡ä
E P = b0
+
x p ?1
N e?
k =1
exp
wk
exp ¡ä
(bk
exp ¡ä
? b0
).
(4)
Here, ¡ä stands for the transferred bases.
3
EYELID EDGE DETECTION AND IDENTIFICATION
We discuss how to detect and identify the four edges for eyelid
motion capture. The four edges have different semantic meanings
representing the double-fold, the upper eyelid, the lower eyelid and
the lower boundary of the bulge, respectively, as shown in fig. 5.
For simplification, we name the four edges as fold edge, top edge,
bottom edge and bulge edge correspondingly. Note that the top edge
and the bottom edge always exist in all facial images (coincident
with each other when the eye is closed), but the fold edge and the
bulge edge may not exist for some eye shapes or poses.
Detecting and identifying the four edges is not a trivial task. Recently, DNN-based edge detection techniques have demonstrated
noticeable improvements in both detection accuracy and performance. However, in this paper, we need to not only detect but also
distinguish the four edges to perform eyelid fitting, which is not
considered in previous edge detection techniques. A naive extension
to identify the four edges is to train four edge detectors, each of
which is individually trained to detect only one of the four edges.
However, for the four edges of each eye region, their relative positions, shapes, and motions are highly correlated. Training four
detectors separately ignores those correlations and does not achieve
good results, as shown in fig. 6(b).
3.1
Network
To exploit the correlation among the four edges, we modify the
holistically-nested edge detection (HED) proposed by Xie et al. [2015]
to train a uniform network that jointly detects and identifies the
four edges. We first formulate the edge detection and identification problem as a multi-channel edge detection problem and then
propose a unified energy metric to jointly consider the four edges
together in the network training, which learns the correlations of
the four edges.
Real-time 3D Eyelids Tracking from Semantic Edges ? 193:5
The HED is based on the VGG-16 net [Simonyan and Zisserman
2014]. It connects five side-output layers to each stage of the VGG16 net. Each side-output layer generates an edge detection result
from the VGG features in the corresponding stage. Each result is
supervised by the ground-truth edge map, and all results are fused
together to generate the final output, which is also supervised by
the ground truth. In our system, since identifying the four edges
is required, we represent our output and also the ground truth as
four-channel binary edge maps, where 1 in each channel stands
for pixels on one of the four edges and 0 is used to label other
pixels. Comparing with the network in the original HED, we use
four convolution kernels, each corresponding to one channel in each
side-output layer and also in the final fused layer. In this manner, we
create a modified network that fits the representation of the output
and jointly achieves detection and identification of the four edges.
3.2
There is a more straightforward way to define L by the individual
detection errors of the four edges:
L(¦× (I, ¦È ), G) =
i=1
To train our network to learn optimal network parameters ¦È , we
formulate the loss function as follows:
arg min(¦Á f L(¦× f (I, ¦È ), G) +
¦È
M
?
¦Á sk L(¦×sk (I, ¦È ), G)).
(5)
k =1
In this function, I and G are the input eye region image and the
four-channel ground truth edge map. ¦× f and ¦×sk stand for the fused
output and the side-outputs of the network. The loss function L
computes the pixel-wise sigmoid cross-entropy loss between an
output edge map and the ground truth. M represents the number of
side-outputs in the network (5 in our experiments). ¦Á f and ¦Á sk are
the weight parameters for the fused loss and the side losses.
We define a unified loss L that integrates the detection and identification errors of the four edges together:
?
L(¦× (I, ¦È ), G) = ?¦Â
logPr(§Õj = 0|I ; ¦È )
j ¡ÊG +
?(1 ? ¦Â)
4 ?
?
i=1 j ¡ÊG ?i
logPr(§Õij = 1|I ; ¦È ),
(6)
?(1 ? ¦Â )
¦Â=
4
?
i
G?
/|G | .
(7)
i=1
i is the set of pixels belonging to the ith edge in the ground truth,
G?
and thus ¦Â stands for the ratio of the number of edge pixels in the
ground truth to the number of all pixels in the eye region image.
G + is the union of all the four set of non-edge pixels in the ground
i . Pr(§Õi = 1|I ; ¦È ) stands for the
truth and each set is defined as G +
j
probability of pixel j belonging to the ith edge in an output, while
Pr(§Õj = 0|I ; ¦È ) stands for the probability of belonging to non-edges.
Pr = ¦Ò (a j ), where ¦Ò is the sigmoid function and a j is the activation
value of pixel j in our DNN. With the definition of eq. (6), minimizing
eq. (5) leads to a network generating desired output. One example
of the input, ground truth, and the output of our network is shown
in fig. 6.
?
logPr(§Õij
(8)
= 1|I ; ¦È )),
j ¡ÊG ?i
where
i
¦Â i = G?
/|G | .
(9)
However, in this definition, the four edges independently contribute
to different terms, thus their correlations are not considered in the
loss. As a consequence, compared with eq. (6), eq. (8) generates
incorrect results as shown in fig. 6(c).
Training
Our training set contains 194 eye region images from 48 identities
and the corresponding four-channel ground truth edge maps (manually labeled). The images of 5 identities are recorded by ourselves
and another 3 are from the Eyediap database [Mora et al. 2014].
These 8 identities contribute 57 facial images, each of which provides two eye region images. For each identity, our data set contains
images with different eyelid poses. The pose changes are caused by
eyelid motions, as well as gaze motions, because eyelids change with
eyeball movements. The remaining 40 identities are collected from
the Internet, each of which has one image. These have no eyelid
pose change, and they contribute 80 training samples in total. Note
that the fold edge and the bulge edge do not exist in some images:
some subjects may not have double eyelids or eye bulges; the top
edge may disappear when subjects look downwards or close their
eyes. In these cases, we do not label fold edges or bulge edges.
In the training, the network is fine-tuned from an initialization of
the pre-trained VGG-16 net model, and we use a standard stochastic
gradient descent algorithm in the training. The parameters in the
training are set as follows: learning rate (1e-6), momentum (0.9),
weight decay (0.0002). The weight parameters ¦Á f and ¦Á sk are all 1.
4
where
j ¡ÊG +i
i
3.3
Loss
4
?
?
(?¦Â i
logPr(§Õij = 0|I ; ¦È )
CURVE-BASED EYELID RECONSTRUCTION
We discuss how to reconstruct the 3D eyelids of a user with our
linear eyelid models and the four edges identified on each recorded
frame. The core idea is to estimate the optimal shape and pose
weights in our linear eyelid models, aiming to minimize the inconsistency between the projected eyelid edges and the real eyelid
edges on the image. To construct the optimization problem, we need
to define the correspondences between the 3D eyelid model and
2D pixels. The edges of the eyelid model are defined by a set of
manually-labeled mesh vertices, called 3D eyelid landmarks. Their
corresponding pixels, called 2D eyelid landmarks, are extracted by
first fitting four polynomial curves to the detected edges on the
edge map (section 4.1), and then determining their locations on the
curves (section 4.2). With the correspondences, we minimize an
energy function that measures the distances between the projected
3D landmarks and their corresponding 2D landmarks (section 4.3).
Note that our eyelid reconstruction method is integrated into a
real-time face tracking system [Wen et al. 2016], which reconstructs
ACM Transactions on Graphics, Vol. 36, No. 6, Article 193. Publication date: November 2017.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- white spot under eyelid and swollen
- typical ophthalmoscopic picture of cherry red spot in an adult with the
- newborn eyelid red
- red spot leaf blotch or measles of peonies
- eyelid abnormalities 4 04 central coast eye vet
- what causes a red eyelid
- 1 getting started 2 eyegaze
- real time 3d eyelids tracking from semantic edges
- children s a e information for parents chalazion moorfields
- eyelid red and swollen
Related searches
- real time options quotes free
- streaming real time stock ticker
- real time stock quotes
- real time quotes stock
- real time stock market ticker
- real time stock ticker free
- real time stock ticker
- real time streaming stock quotes
- yahoo finance real time quotes
- free real time stock charts
- free real time stock quotes streaming
- real time stock charts free