demonstration

Towards Image Design Space Exploration in Spreadsheets with LLM Formulae

Authors:

J.D. Zamfirescu-Pereira,

Shm Garanganao Almeda,

Bjoern HartmannAuthors Info & Claims

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Article No.: 82, Pages 1 - 3

https://doi.org/10.1145/3586182.3615790

Published: 29 October 2023 Publication History

Abstract

Users of Text-to-Image (TTI) models like DALL•E and Stable Diffusion typically engage in a lot of iteration, exploring a design space with two main inputs: (1) prompt text spanning image content and style; and (2) stochastic (e.g., random seeds) and other opaque (e.g., classifier-free guidance) variables. Here, we demo an early prototype interface using a spreadsheet metaphor to enable exploration and display of multiple input changes simultaneously, and affording prompt-crafting using spreadsheet formula construction. New LLM-based functions aid rapid exploration of the prompt text input space, by generating new variations on existing prompts and context-relevant lists of prompt keyword options.

References

[1]

Maneesh Agrawala. 2023. Unpredictable Black Boxes are Terrible Interfaces. https://magrawala.substack.com/p/unpredictable-black-boxes-are-terrible

[2]

Pol Baladas Gerard Serra, Oriol Domingo. 2022. A programmable interface for creative exploration. Machine Learning for Creativity and Design Workshop at the 36th Conference on Neural Information Processing Systems (NeurIPS 2022) (December 2022). https://neuripscreativityworkshop.github.io/2022/papers/ml4cd2022_paper19.pdf

[3]

Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R. Klemmer. 2008. Design as Exploration: Creating Interface Alternatives through Parallel Authoring and Runtime Tuning. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST ’08). Association for Computing Machinery, New York, NY, USA, 91–100. https://doi.org/10.1145/1449715.1449732

Digital Library

[4]

Jeffrey Heer, Jock Mackinlay, Chris Stolte, and Maneesh Agrawala. 2008. Graphical histories for visualization: Supporting analysis, communication, and evaluation. IEEE transactions on visualization and computer graphics 14, 6 (2008), 1189–1196.

[5]

Chinmay Kulkarni, Stefania Druga, Minsuk Chang, Alex Fiannaca, Carrie Cai, and Michael Terry. 2023. A Word is Worth a Thousand Pictures: Prompts as AI Design Material. arXiv preprint arXiv:2303.12647 (2023).

[6]

Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R Klemmer. 2010. Designing with interactive example galleries. In Proceedings of the SIGCHI conference on human factors in computing systems. 2257–2266.

Digital Library

[7]

Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 384, 23 pages. https://doi.org/10.1145/3491102.3501825

Digital Library

[8]

Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2022. 3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows. arxiv:2210.11603 [cs.HC]

[9]

Aran Lunzer and Kasper Hornbæk. 2008. Subjunctive Interfaces: Extending Applications to Support Parallel Setup, Viewing and Control of Alternative Scenarios. ACM Trans. Comput.-Hum. Interact. 14, 4, Article 17 (jan 2008), 44 pages. https://doi.org/10.1145/1314683.1314685

Digital Library

[10]

J. Marks, B. Andalman, P. A. Beardsley, W. Freeman, S. Gibson, J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall, J. Seims, and S. Shieber. 1997. Design Galleries: A General Approach to Setting Parameters for Computer Graphics and Animation. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 389–400. https://doi.org/10.1145/258734.258887

Digital Library

[11]

proxima�centauri b, surea.i, Stephen Young, and Encyclopedia Erratica. 2023. Image Synthesis Style Studies Database by parrotzone.art. https://web.archive.org/web/20230325013519/https://docs.google.com/spreadsheets/d/14xTqtuV3BuKDNhLotB_d1aFlBGnDJOY0BRXJ8-86GpA/edit#gid=0

[12]

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. arxiv:2204.06125 [cs.CV]

[13]

Eric Rawn, Jingyi Li, Eric Paulos, and Sarah Chasins. 2023. Understanding Version Control as Material Interaction with Quickpose. In Proceedings of the SIGCHI conference on human factors in computing systems.

[14]

Daniel Ritchie, Ankita Arvind Kejriwal, and Scott R Klemmer. 2011. d. tour: Style-based exploration of design example galleries. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 165–174.

Digital Library

[15]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2022), 10684–10695.

[16]

Robert F. Woodbury and Andrew L. Burrow. 2006. A typology of design space explorers. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 20, 2 (2006), 143–153. https://doi.org/10.1017/S0890060406060136

Digital Library

[17]

JD Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. (2023).

[18]

J.D. Zamfirescu-Pereira, Richmond Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems (CHI’23).

Digital Library

Cited By

Zhou MZhang DYou WYu ZWu YPan CLiu HLao TChen P(2024)StyleFactory: Towards Better Style Alignment in Image Creation through Style-Strength-Based Control and EvaluationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676370(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676370
Zhong RShin DMeza RKlasnja PColusso LHsieh G(2024)AI-Assisted Causal Pathway Diagram for Human-Centered DesignProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642179(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642179

Index Terms

Towards Image Design Space Exploration in Spreadsheets with LLM Formulae
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms

Recommendations

Fashioning Creative Expertise with Generative AI: Graphical Interfaces for Design Space Exploration Better Support Ideation Than Text Prompts
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

This paper investigates the potential impact of deep generative models on the work of creative professionals. We argue that current generative modeling tools lack critical features that would make them useful creativity support tools, and introduce our ...
Fashioning the Future: Unlocking the Creative Potential of Deep Generative Models for Design Space Exploration
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

This paper investigates the potential impact of deep generative models on the work of creative professionals, specifically focusing on fashion design. We argue that current generative modeling tools lack critical features that would make them useful ...
Design space exploration acceleration through operation clustering

This paper presents a clustering method called clustering design space exploration (CDS-ExpA) to accelerate the architectural exploration of behavioral descriptions in C and SystemC. The trade-offs between faster exploration versus optimality of results ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

October 2023

424 pages

ISBN:9798400700965

DOI:10.1145/3586182

Editors:
Sean Follmer
Stanford University, USA
,
Jeff Han,
J�rgen Steimle
Saarland University, Germany
,
Nathalie Henry Riche
Microsoft Research, USA

Copyright � 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Funding Sources

USAF/DARPA SDCPS Program

Conference

UIST '23

Sponsor:

UIST '23: The 36th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 1, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)199
Downloads (Last 6 weeks)14

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou MZhang DYou WYu ZWu YPan CLiu HLao TChen P(2024)StyleFactory: Towards Better Style Alignment in Image Creation through Style-Strength-Based Control and EvaluationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676370(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676370
Zhong RShin DMeza RKlasnja PColusso LHsieh G(2024)AI-Assisted Causal Pathway Diagram for Human-Centered DesignProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642179(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642179

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents