Publications

Recent success suggests that deep neural control networks are likely to be a key component of self-driving vehicles. These networks are trained on large datasets to imitate human actions, but they lack semantic understanding of image contents. This makes them brittle and potentially unsafe in situations that do not match training data. Here, we propose to address this issue by augmenting training data with natural language advice from a human. Advice includes guidance about what to do and where to attend. We present the first step toward advice giving, where we train an end-to-end vehicle controller that accepts advice. The controller adapts the way it attends to the scene (visual attention) and the control (steering and speed). Attention mechanisms tie controller behavior to salient objects in the advice. We evaluate our model on a novel advisable driving dataset with manually annotated human-to-vehicle advice called Honda Research Institute-Advice Dataset (HAD). We show that taking advice improves the performance of the end-to-end network, while the network cues on a variety of visual features that are provided by advice. The dataset is available at https://usa.honda-ri.com/HAD.

An entry from the Cambridge Structural Database, the world’s repository for small molecule crystal structures. The entry contains experimental data from a crystal diffraction study. The deposited dataset for this entry is freely available from the CCDC and typically includes 3D coordinates, cell parameters, space group, experimental conditions and quality measures.

An entry from the Cambridge Structural Database, the world’s repository for small molecule crystal structures. The entry contains experimental data from a crystal diffraction study. The deposited dataset for this entry is freely available from the CCDC and typically includes 3D coordinates, cell parameters, space group, experimental conditions and quality measures.

We study a modular approach to tackle long-horizon mobile manipulation tasks for object rearrangement, which decomposes a full task into a sequence of subtasks. To tackle the entire task, prior work chains multiple stationary manipulation skills with a point-goal navigation skill, which are learned individually on subtasks. Although more effective than monolithic end-to-end RL policies, this framework suffers from compounding errors in skill chaining, e.g., navigating to a bad location where a stationary manipulation skill can not reach its target to manipulate. To this end, we propose that the manipulation skills should include mobility to have flexibility in interacting with the target object from multiple locations and at the same time the navigation skill could have multiple end points which lead to successful manipulation. We operationalize these ideas by implementing mobile manipulation skills rather than stationary ones and training a navigation skill trained with region goal instead of point goal. We evaluate our multi-skill mobile manipulation method M3 on 3 challenging long-horizon mobile manipulation tasks in the Home Assistant Benchmark (HAB), and show superior performance as compared to the baselines.

Abstract The unfolded protein response in the endoplasmic reticulum (UPR ER ) is involved in a number of metabolic diseases, including non-alcoholic fatty liver disease. Here, we characterize the UPR ER induced metabolic changes in mouse liver through in vivo metabolic labeling and mass spectrometric analysis of proteome and lipid fluxes. We induced ER stress in vivo via tunicamycin treatment and measured rates of proteome-wide protein synthesis, de novo lipogenesis and cholesterol synthesis serially over a three-day period, thereby generating a metabolic “signature” of the UPR ER over time. Synthesis of most proteins was suppressed under ER stress conditions, including proteins involved in lipogenesis, consistent with reduced de novo lipogenesis at 48 and 72 hours. Electron microscopy revealed striking morphological changes to ER and H&E staining showed lipid droplet enriched livers under ER stress. Pre-labeling of adipose tissue prior to ER stress induction revealed mobilization of lipids from adipose to the liver. Interestingly, the source of these lipids was uptake of free fatty acids, not whole triglycerides or phospholipids from lipoproteins, as demonstrated by replacement of the triglyceride-glycerol moiety in liver concurrently with increased incorporation of labeled palmitate from adipose. We also induced ER stress by a high-fat diet and observed similar metabolic flux signatures, suggesting that this mechanism may play a role in the progression of fatty liver disease. This flux-based approach provides a powerful tool to identify novel regulators of ER stress and potential targets for pharmacological intervention.

Objects manipulated by the hand (i.e., manipulanda) are particularly challenging to reconstruct from Internet videos. Not only does the hand occlude much of the object, but also the object is often only visible in a small number of image pixels. At the same time, two strong anchors emerge in this setting: (1) estimated 3D hands help disambiguate the location and scale of the object, and (2) the set of manipulanda is small relative to all possible objects. With these insights in mind, we present a scalable paradigm for hand-held object reconstruction that builds on recent breakthroughs in large language/vision models and 3D object datasets. Given a monocular RGB video, we aim to reconstruct hand-held object geometry in 3D, over time. In order to obtain the best performing single frame model, we first present MCC-Hand-Object (MCC-HO), which jointly reconstructs hand and object geometry given a single RGB image and inferred 3D hand as inputs. Subsequently, we prompt a text-to-3D generative model using GPT-4(V) to retrieve a 3D object model that matches the object in the image(s); we call this alignment Retrieval-Augmented Reconstruction (RAR). RAR provides unified object geometry across all frames, and the result is rigidly aligned with both the input images and 3D MCC-HO observations in a temporally consistent manner. Experiments demonstrate that our approach achieves state-of-the-art performance on lab and Internet image/video datasets. We make our code and models available on the project website: https://janehwu.github.io/mcc-ho

Hot electron flow generated on colloid platinum nanoparticles during exothermic catalytic carbon monoxide oxidation was directly detected with Au/TiO2 diodes. Although Au/TiO2 diodes are not catalytically active, platinum nanoparticles on Au/TiO2 exhibit both chemicurrent and catalytic turnover rate. Hot electrons are generated on the surface of the metal nanoparticles and go over the Schottky energy barrier between Au and TiO2. The continuous Au layer ensures that the metal nanoparticles are electrically connected to the device. The overall thickness of the metal assembly (nanoparticles and Au thin film) is comparable to the mean free path of hot electrons, resulting in ballistic transport through the metal. The chemicurrent and chemical reactivity of nanoparticles with citrate, hexadecylamine, hexadecylthiol, and TTAB (tetradecyltrimethylammonium bromide) capping agents were measured during catalytic CO oxidation at pressures of 100 Torr O2 and 40 Torr CO at 373-513 K. We found that chemicurrent yield varies with each capping agent but always decreases with increasing temperature. We suggest that this inverse temperature dependence is associated with the influence of charging effects due to the organic capping layer during hot electron transport through the metal-oxide interface.