Towards Pervasive Human-Centric Cyber-Physical Systems

At the intersection of design, artificial intelligence, and cyber-physical systems lies a frontier of innovation that promises to revolutionize our built environment and daily lives. My research agenda explores this intersection, focusing on the development of intelligent, adaptive environments and agents that seamlessly integrate with human needs and behaviors. This work forms part of the Omicron project, investigating how sensor networks, actuators, and AI can create dynamically responsive environments.

Central to this research is a three-axis model:

  1. Information Filtration Level: Spanning unfiltered personal spaces to highly filtered public areas.
  2. Intelligence Hierarchy: Ranging from basic asset integration to fully autonomous systems.
  3. Integration Layer: Encompassing bio-integrated devices to ambient intelligence networks.
Omicron Layers

This model guides our exploration of how varying levels of technological integration can serve diverse contexts and human needs. A key outcome will be a specification for human-centric automated systems in built environments, drawing inspiration from Industry 4.0 & 5.0 and the IEC RAMI4.0 standard.

My work proposes a multi-modal approach to interaction, combining traditional interfaces with emerging technologies such as natural language processing, gesture recognition, and augmented reality. This approach is grounded in the concept of contextualized agent systems integrated into our environments, necessitating seamless cooperation across multiple levels and modalities.

A focal point of my research is the development of an open model of computation allowing for highly secure interaction across layers of a technological mesh. This model incorporates elements of the Actor Model for independent node communication, alongside a blockchain-inspired ledger system for real-time data sharing and access control. This dual communication system enables both traditional message-based interaction and real-time access to relevant data changes across the mesh, facilitating the critical interplay between implanted bio-integrated devices, adaptive environments, and AI assistants.

Drawing inspiration from Industry 4.0 and 5.0 concepts, particularly the notion of Cyber-Physical Systems, this research seeks to extend the principles of connected systems beyond industrial applications into everyday living spaces. The goal is to create a seamless integration of sensors, machines, and IT systems throughout the value chain of architectural design and inhabitation.

Privacy and user ownership of data are paramount in this research. By focusing on decentralized compute models and exploring technologies like private cloud compute and decentralized social networks, I aim to ensure that users maintain control over their personal information while benefiting from deeply integrated technologies.

This research also investigates cross-contextual, environment-aware agents capable of operating across various types of spaces while maintaining appropriate privacy boundaries. This includes developing protocols for agent-environment communication and context switching, as well as exploring the architectural implications of spaces that can adapt to user preferences communicated via personal agents.

Ultimately, this research aims to redefine the built environment as an intelligent system of asynchronous, distributed compute nodes that communicates with other intelligent systems. By conceptualizing our spaces and environments as partially robotic, embodied intelligent agents, we open new avenues for designing environments that can sense, respond, and adapt to user needs.

Through this multifaceted research agenda, I seek to push the boundaries of design and technology, creating spaces that not only house advanced technologies but actively participate in enhancing human experiences and capabilities. This work has the potential to significantly impact how we conceive, design, and interact with our built environment in the age of pervasive computing and artificial intelligence.

References

  1. [1] Vitor Alcacer and Virgilio Cruz-Machado. Scanning the industry 4.0: A literature review on technologies for manufacturing systems. Engineering science and technology, an international journal, 22(3):899–919, 2019.
  2. [2] Bahareh Barati, Elvin Karana, Sylvia Pont, and Tim Van Dortmont. Living light interfaces—an exploration of bioluminescence aesthetics. In Proceedings of the 2021 ACM Designing Interactive Systems Conference, pages 1215–1229.
  3. [3] Fiona Bell, Joshua Coffie, and Mirela Alistar. Bio-digital calendar: Attuning to nonhuman temporalities for multispecies understanding. In Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction, pages 1–15.
  4. [4] Fiona Bell, Alice Hong, Andreea Danielescu, Aditi Maheshwari, Ben Greenspan, Hiroshi Ishii, Laura Devendorf, and Mirela Alistar. Self-destaining textiles: Designing interactive systems with fabric, stains and light. ACM.
  5. [5] Fiona Bell, Alice Hong, Andreea Danielescu, Aditi Maheshwari, Ben Greenspan, Hiroshi Ishii, Laura Devendorf, and Mirela Alistar. Self-destaining textiles: Designing interactive systems with fabric, stains and light. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–12.
  6. [6] Fiona Bell, Michelle Ramsahoye, Joshua Coffie, Julia Tung, and Mirela Alistar. µme: Exploring the human microbiome as an intimate material for living interfaces. In Proceedings of the 2023 ACM Designing Interactive Systems Conference, pages 2019–2033.
  7. [7] Lucas Bellaiche, Adam Haar Horowitz, Mason McClay, Ryan Bottary, Dan Denis, Christina Chen, Pattie Maes, and Paul Seli. Targeted dream incubation at a distance: the development of a remote and sensor-free tool for incubating hypnagogic dreams and mind-wandering. Frontiers in Sleep, 3:1258345, 2024.
  8. [8] Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024.
  9. [9] Zana Bu¸cinca, Siddharth Swaroop, Amanda E Paluch, Susan A Murphy, and Krzysztof Z Gajos. Towards optimizing human-centric objectives in ai-assisted decision-making with offline reinforcement learning. arXiv preprint arXiv:2403.05911, 2024.
  10. [10] Nadire Cavus, Oluwafemi Ayotunde Oke, and Jamilu Maipan-uku Yahaya. Brain-Computer Interfaces: High-Tech Race to Merge Minds and Machines, pages 3–19. Springer, 2023.
  11. [11] Michelle Chang, Chenyi Shen, Aditi Maheshwari, Andreea Danielescu, and Lining Yao. Patterns and opportunities for the design of human-plant interaction. In Proceedings of the 2022 ACM Designing Interactive Systems Conference, pages 925–948.
  12. [12] Chun-Wei Chiang, Zhuoran Lu, Zhuoyan Li, and Ming Yin. Enhancing ai-assisted group decision making through llm-powered devil’s advocate. ACM.
  13. [13] Valdemar Danry, Pat Pataranutaporn, Adam Haar Horowitz, Paul Strohmeier, Josh Andres, Rakesh Patibanda, Zhuying Li, Takuto Nakamura, Jun Nishida, and Pedro Lopes. Do cyborgs dream of electric limbs? experiential factors in human-computer integration design and evaluation. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–6.
  14. [14] Kevin Dunnell, Gauri Agarwal, Pat Pataranutaporn, Andrew Lippman, and Pattie Maes. Ai-generated media for exploring alternate realities. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–8.
  15. [15] Seliem El-Sayed, Canfer Akbulut, Amanda McCroskery, Geoff Keeling, Zachary Kenton, Zaria Jalan, Nahema Marchal, Arianna Manzini, Toby Shevlane, and Shannon Vallor. A mechanism-based approach to mitigating harms from persuasive generative ai. arXiv preprint arXiv:2404.15058, 2024.
  16. [16] Jack Forman, Pat Pataranutaporn, Phillip Gough, Raphael Kim, Fiona Bell, Netta Ofer, Jasmine Lu, Angela Vujic, Muqing Bai, and Pattie Maes. Living bits and radical aminos: A workshop on bio-digital interfaces for human-computer interaction. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–7.
  17. [17] Olga Gelonch, Mireia Ribera, N´uria Codern-Bov´e, S´ılvia Ramos, Maria Quintana, Gloria Chico, Noem´ı Cerulla, Paula Lafarga, Petia Radeva, and Maite Garolera. Acceptability of a lifelogging wearable camera in older adults with mild cognitive impairment: a mixed-method study. BMC geriatrics, 19:1–10, 2019.
  18. [18] Phillip Gough, Jack Forman, Pat Pataranutaporn, Leigh-Anne Hepburn, Carolina Ramirez-Figueroa, Clare Cooper, Angela Vujic, David Sun Kong, Raphael Kim, and Pattie Maes. Speculating on biodesign in the future home. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–5.
  19. [19] Phillip Gough, Praneeth Bimsara Perera, Michael A Kertesz, and Anusha Withana. Design, mould, grow!: A fabrication pipeline for growing 3d designs using myco-materials. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–15.
  20. [20] Phillip Gough, Soojeong Yoo, Martin Tomitsch, and Naseem Ahmadpour. Applying bioaffordances through an inquiry-based model: a literature review of interactive biodesign. International Journal of Human–Computer Interaction, 37(17):1583–1597, 2021.
  21. [21] Declan Grabb, Max Lamparth, and Nina Vasan. Risks from language models for automated mental healthcare: Ethics and structure for implementation. 2024.
  22. [22] Eduard Georges Groutars, Carmen Clarice Risseeuw, Colin Ingham, Raditijo Hamidjaja, Willemijn S Elkhuizen, Sylvia C Pont, and Elvin Karana. Flavorium: an exploration of flavobacteria’s living aesthetics for living color interfaces. In Proceedings of the 2022 CHI conference on human factors in computing systems, pages 1–19.
  23. [23] Alicia Guo, Pat Pataranutaporn, and Pattie Maes. Exploring the impact of ai value alignment in collaborative ideation: Effects on perception, ownership, and output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–11.
  24. [24] Martin Hankel and Bosch Rexroth. The reference architectural model industrie 4.0 (rami 4.0). Zvei, 2(2):4–9, 2015.
  25. [25] Chandan Hunsur Ravikumar, Paskorn Muangphrom, Pat Pataranutaporn, and Werasak Surareungchai. Future Perspective of Nanobiomaterials in Human Health Care, pages 741–761. Springer, 2023.
  26. [26] Lujain Ibrahim, Luc Rocher, and Ana Valdivia. Characterizing and modeling harms from interactions with design patterns in ai interfaces. arXiv preprint arXiv:2404.11370, 2024.
  27. [27] ISO ISO. Open systems interconnection-basic reference model. ISO/TC, 97:7498–1, 1992.
  28. [28] Waheb A Jabbar, Tee Kok Kian, Roshahliza M Ramli, Siti Nabila Zubir, Nurthaqifah SM Zamrizaman, Mohammed Balfaqih, Vladimir Shepelev, and Soltan Alharbi. Design and fabrication of smart home with internet of things enabled automation system. IEEE access, 7:144059–144074, 2019.
  29. [29] Elvin Karana, Bahareh Barati, and Elisa Giaccardi. Living artefacts: Conceptualizing livingness as a material quality in everyday artefacts. International Journal of Design, 14(3), 2020.
  30. [30] Kimon Kieslich, Nicholas Diakopoulos, and Natali Helberger. Anticipating impacts: using large-scale scenario-writing to explore diverse implications of generative ai in the news environment. AI and Ethics, pages 1–23, 2024.
  31. [31] Kimon Kieslich, Natali Helberger, and Nicholas Diakopoulos. My future with my chatbot: A scenario-driven, user-centric approach to anticipating ai impacts. arXiv preprint arXiv:2401.14533, 2024.
  32. [32] Raphael Kim, Pat Pataranutaporn, Jack Forman, Seung Ah Lee, Ingmar H Riedel-Kruse, Mirela Alistar, Eldy S Lazaro Vasquez, Katia Vega, Roland Van Dierendonck, and Gilad Gome. Microbe-hci: Introduction and directions for growth. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–4.
  33. [33] Raphael Kim, Clarice Risseeuw, Eduard Georges Groutars, and Elvin Karana. Surfacing livingness in microbial displays: A design taxonomy for hci. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–21.
  34. [34] Mik Lamming and Denis Bohm. Specs: personal pervasive systems. Computer, 36(6):109–111, 2003.
  35. [35] Yezdi Lashkari, Max Metral, and Pattie Maes. Collaborative interface agents. In AAAI, volume 94, pages 444–449. Citeseer.
  36. [36] Seung Ah Lee and Ingmar H Riedel-Kruse. Micro-hbi: human-biology interaction with living cells, viruses, and molecules. Frontiers in Computer Science, 4:849887, 2022.
  37. [37] Joanne Leong, Pat Pataranutaporn, Valdemar Danry, Florian Perteneder, Yaoli Mao, and Pattie Maes. Putting things into context: Generative ai-enabled context personalization for vocabulary learning improves learning motivation. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–15.
  38. [38] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024.
  39. [39] Xin Li, Ke Rong, and Xinwei Shi. Situating artificial intelligence in organization: A human-machine relationship perspective. Journal of Digital Economy, 2:330–335, 2023.
  40. [40] Lauren Lin and Duri Long. Generative ai futures: A speculative design exploration. In Proceedings of the 15th Conference on Creativity and Cognition, pages 380–383.
  41. [41] Jasmine Lu and Pedro Lopes. Integrating living organisms in devices to implement care-based interactions. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–13.
  42. [42] Yang Lu. Industry 4.0: A survey on technologies, applications and open research issues. Journal of industrial information integration, 6:1–10, 2017.
  43. [43] Shuai Ma, Qiaoyi Chen, Xinru Wang, Chengbo Zheng, Zhenhui Peng, Ming Yin, and Xiaojuan Ma. Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making. arXiv preprint arXiv:2403.16812, 2024.
  44. [44] Shuai Ma, Chenyi Zhang, Xinru Wang, Xiaojuan Ma, and Ming Yin. Beyond recommender: An exploratory study of the effects of different ai roles in ai-assisted decision making. arXiv preprint arXiv:2403.01791, 2024.
  45. [45] Praveen Kumar Reddy Maddikunta, Quoc-Viet Pham, B Prabadevi, Natarajan Deepa, Kapal Dev, Thippa Reddy Gadekallu, Rukhsana Ruby, and Madhusanka Liyanage. Industry 5.0: A survey on enabling technologies and potential applications. Journal of Industrial Information Integration, 26:100257, 2022.
  46. [46] Pattie Maes. Modeling adaptive autonomous agents. Artificial life, 1(1 2):135–162, 1993.
  47. [47] Pattie Maes. Artificial life meets entertainment: lifelike autonomous agents. Communications of the ACM, 38(11):108–114, 1995.
  48. [48] Pattie Maes and Robyn Kozierok. Learning interface agents. In AAAI, volume 93, pages 459–465.
  49. [49] Clara Mancini, Yvonne Rogers, Arosha K. Bandara, Tony Coe, Lukasz Jedrzejczyk, Adam N. Joinson, Blaine A. Price, Keerthi Thomas, and Bashar Nuseibeh. Contravision: Exploring users’ reactions to futuristic technology. ACM.
  50. [50] Holly McQuillan and Elvin Karana. Conformal, seamless, sustainable: Multimorphic textile-forms as a material-driven design approach for hci. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–19.
  51. [51] Pranav Mistry and Pattie Maes. SixthSense: a wearable gestural interface, pages 85–85. 2009.
  52. [52] Dipesh Niraula, Kyle C. Cuneo, Ivo D. Dinov, Brian D. Gonzalez, Jamalina B. Jamaluddin, Jionghua Jin, Yi Luo, Martha M. Matuszak, Randall K. Ten Haken, Alex K. Bryant, Thomas J. Dilling, Michael P. Dykstra, Jessica M. Frakes, Casey L. Liveringhouse, Sean R. Miller, Matthew N. Mills, Russell F. Palm, Samuel N. Regan, Anupam Rishi, Javier F. Torres-Roca, Hsiang-Hsuan Michael Yu, and Issam El Naqa. Intricacies of human-ai interaction in dynamic decision-making for precision oncology: A case study in response-adaptive radiotherapy. (Judy), 2024.
  53. [53] Netta Ofer and Mirela Alistar. Felt experiences with kombucha scoby: Exploring first-person perspectives with living matter. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–18.
  54. [54] Netta Ofer, Fiona Bell, and Mirela Alistar. Designing direct interactions with bioluminescent algae. In Proceedings of the 2021 ACM Designing Interactive Systems Conference, pages 1230–1241.
  55. [55] Pat Pataranutaporn, Valdemar Danry, Joanne Leong, Parinya Punpongsanon, Dan Novy, Pattie Maes, and Misha Sra. Ai-generated characters for supporting personalized learning and well-being. Nature Machine Intelligence, 3(12):1013–1022, 2021.
  56. [56] Pat Pataranutaporn, Todd Ingalis, and Ed Finn. Biological hci: Toward integrative interfaces between people, computer, and biological materials. CHI 2018, 2018.
  57. [57] Pat Pataranutaporn, Ruby Liu, Ed Finn, and Pattie Maes. Influencing human–ai interaction by priming beliefs about ai can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5(10):1076–1086, 2023.
  58. [58] Pat Pataranutaporn, Angela Vujic, David S. Kong, Pattie Maes, and Misha Sra. Living bits : Opportunities and challenges for integrating living microorganisms in human-computer interaction. ACM.
  59. [59] Pat Pataranutaporn, Kavin Winson, Peggy Yin, Auttasak Lapapirojn, Pichayoot Ouppaphan, Monchai Lertsutthiwong, Pattie Maes, and Hal Hershfield. Future you: A conversation with an ai-generated future self reduces anxiety, negative emotions, and increases future self-continuity. arXiv e-prints, page arXiv: 2405.12514, 2024.
  60. [60] Daniel Pillis, Pat Pataranutaporn, Pattie Maes, and Misha Sra. Ai comes out of the closet: Using ai-generated virtual characters to help individuals practice lgbtqia+ advocacy. ACM.
  61. [61] Clarice Risseeuw, Holly Mcquillan, Joana Martins, and Elvin Karana. (re) activate,(re) direct,(re) arrange: Exploring the design space of direct interactions with flavobacteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–18.
  62. [62] Michael Rußmann, Markus Lorenz, Philipp Gerbert, Manuela Waldner, Jan Justus, Pascal Engel, and Michael Harnisch. Industry 4.0: The future of productivity and growth in manufacturing industries. Boston consulting group, 9(1):54–89, 2015.
  63. [63] Olivia Seow, Cedric Honnet, Simon Perrault, and Hiroshi Ishii. Pudica: A framework for designing augmented human-flora interaction. In Proceedings of the Augmented Humans International Conference 2022, pages 40–45.
  64. [64] Katherine Wei Song and Eric Paulos. Vim: Customizable, decomposable electrical energy storage. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–18.
  65. [65] Wil V Srubar. Engineered living materials: taxonomies and emerging trends. Trends in biotechnology, 39(6):574–583, 2021.
  66. [66] Yudai Tanaka, Angela Vujic, Pattie Maes, Robert JK Jacob, Olaf Blanke, Sho Nakagome, and Pedro Lopes. Neurochi: Are we prepared for the integration of the brain with computing? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–5.
  67. [67] Angela Vujic, Shreyas Nisal, and Pattie Maes. Joie: a joy-based brain-computer interface (bci). In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–14.
  68. [68] Daniel Wangpraseurt, Shangting You, Yazhi Sun, and Shaochen Chen. Biomimetic 3d living materials powered by microorganisms. Trends in Biotechnology, 40(7):843–857, 2022.
  69. [69] Stephan Weyer, Mathias Schmitt, Moritz Ohmer, and Dominic Gorecky. Towards industry 4.0-standardization as the crucial challenge for highly modular, multi-vendor production systems. Ifac-Papersonline, 48(3):579–584, 2015.
  70. [70] Wazeer Deen Zulfikar, Samantha Chan, and Pattie Maes. Memoro: Using large language models to realize a concise interface for real-time memory augmentation. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–18, 2024.