You complete me is a Chrome Extension that replaces the functional Gmail replies auto-complete suggestions by romantic notes. You and your machine complete each other, now it can complete you in a more loving way <3
We are already used to machines completing our sentences. In our Google searches…
… in our phones…
… and now also when writing e-mails.
In Portuguese, English… this is indeed great tool that increased my productivity and ability to answer people.
At the same time though, it is a bit weird that my gmail is able to help me answer things that, yes, i could definitely answer myself, in such a remarkable, assertive way.
And is very interesting to think how this changes the way that I communicate with people. With clients, co-workers, professors, friends. Isn’t it weird that a machine now has the amazing capacity to answer for us? Maybe even replace our approval or interaction at some point soon.
With this in mind, and inviting us to rethink our interaction with auto complete and with the way computers are completing us in general, I created this ironic piece “You Complete Me” as this Chrome Extension that “hacks” the natural autocomplete, changing the messaging to cheesy romantic sentences.
For now the app does not take the e-mail text in consideration, it is just a random array of scentences scraped from the amazing websites Love Quotes For Him and Her but the idea is to add Watsons tone analyzer and customize the reponses accordingly.
The way we understand the internet today is blurred by the idea that the virtual world is indeed only virtual and omnipresent. And this is a problem: when people believe blindly in something, we delegate a lot of power to the structures that make this network possible. Behind the cloud is the result of my research on the boundaries of the internet and the infrastructure of natural resources, human labor and data that lies behind it. In sum, this project addresses and personifies this “Myth of the cloud”, aiming to make this information more accessible and interactive in order to help users question the status quo of the technology that shapes the way we behave today.
Click in the gif below to access and interact with the project.
Below I will describe about the process and references that led me to create this experience.
It all started with a different question…
For the past couple of years I have been working in the tech industry in different roles, towards the path that led me to the designer and technologist that I am today. I’ve always been passionate about Human Computer Interaction and amazed by the democratic aspect of it. When creating an UX experience, usually for websites or mobile apps, the first thing I think is: “Who am I designing for?”, and do my research to make the best interaction possible accordingly.
But…what about who I am NOT designing for?— what about the people that don’t want to be or that are not online today? Have I ever really researched about them? Isn’t it important that as a designer, and mainly as an optimist person that believes in the potential of the internet as a democratic tool, to really understand the limitations and boundaries of this technology? I thought so, and this idea was stuck in my head.
Consequently, I decided to take this question as the starting point of my research during this semester.
By starting to research through reading a couple of UNICEF and World Economic Forum reports on internet penetration reach and also by talking with fellow students at ITP that are very in depth with the research around data privacy, I started to map the world outside the mainstream web, trying to identify the existing boundaries of user presence in what we understand as the internet today.
This mapping exercise made me realize the two main extremes in these spectrum:
The reality of not being online
Basically 50% of the world is not online. Even though this is very well known, this number is very striking for me.
We act as all the information available online represents all the information on the world. And this is not true. We do map most regions and content creation and access is very well spread around this connected half of the planet. But, besides the information we have access to is mainly shaped by the 5 big platforms/companies players in the market (Amazon, Apple, Google, Facebook, Microsoft), it is also very biased as the other half of the planet doesn’t take part in this virtual world.
Of course internet access and penetration are a result of other gaps, such as economic, education, health, and so on – which is a reflect of these countries and regions being also excluded in these other spheres of the global society. It is not a novelty that we live in an extreme unequal system. As Eleanor Saitta puts
“All technical problems of sufficient scope or impact are actually political problems first”.
What is interesting for me, is that this fact is not highlighted when we think about the computation ideal of big data studies and when we create this myth that we can replicate a perfect model of our planet when analyzing online behavior — we can indeed, but first we have to acknowledge that this accessibility gap also exists online.
The ideal of not being online
On the other extreme of this unequal world, we have the rising awareness in data privacy, and the ideal of changing what the mainstream online is. The idea of a decentralized internet where information is not stored in the main data centers of the companies mentioned above and that also can live free from those platforms. The idea of creating open source decentralized products and networks that go beyond of what we understand of the mainstream World Wide Web, the regulations on data are also extremely important. Those issues and laws are usually raised in countries where internet access is widespread, mainly in Europe and North America, which again, very much relates to the quote from Eleanor Saitta. As long as we live in this capitalist system, it seems natural that the environment that we create online will also replicate this reality.
Understanding that these extremes, I decided to make some actual maps where I could visualize them. Hoping that they could give me some insight on my research. Therefore, I dived into these reports data, and created the maps below.
The infrastructure of the internet
Parallel to this research, and as a reflect of understanding that these extremes in the virtual world are a reflect of our real world, I decided to explore why we as creators and users perceive this as separate realities while they are so much of the same. Where does these disconnection between the physical and the virtual begins?
One amazing reference was Kate Crawford’s work, mainly on this article: Anatomy of an AI system.The author tries to map every aspect triggered from the moment an user asks their Amazon Echo to turn on the lights. Everything then, from the mined resources, to the workers, data centers, the data collected from that user too, and the e-waste that the amazon echo material will soon be, are also triggered and unveiled in the text and in the infograph created to illustrate it.
Also, a great reference at this point was the book the New Dark Age, by James Bridle, where he mentions the connections between submarine internet cables and the colonial routes from the colonial period. Which – by going back to my map-dataviz experiments started to play a bit with.
From this part of the research on, I understood that maybe “Who is not online?” wasn’t a good question for my project anymore, but something that related me more towards the search of mapping what was the infrastructure of being online meant.
What’s Behind the cloud?
Why do we relate with this virtual world we created as it is detached from our physical reality? Why do we see technology – as James Bindle puts – “…this leveling force, that makes us more equal and allow us to make better, more equitable decisions”? Why do we believe so blindly in the algorithms and see it as something magical?
Internet as a Hyperobject
The idea of computation thinking and the scope that internet took as an Hyperobject, enabled by our systems and by the reinforcement of the way platforms today are designed to be – minimalist, functional and wireless – created this faith in the network that it is this magical far away place where all this stuff we use just happens and comes back to us. But the reality is that it is super physical, we work creating it, it needs natural resources, energy, data, and it generates chaos, and waste and heat… and so much more.
It is indeed overwhelming to think that in less than two decades we went from the physicality of having to dial to a line…
…to this Myth of the cloud we created today.
The term “cloud computing” was popularized with Amazon when the company released its Elastic Compute Cloud product in 2006.
The main enabling technology for cloud computing is this said ‘virtualization’, when it separates a physical computing device into one or more devices, each of which can be easily used and managed to perform computing tasks. With operating system–level virtualization essentially creating a scalable system of multiple independent computing devices, idle computing resources can be allocated and used more efficiently.
In this sense, the idea of it being virtual is that, instead of stored in a local data base, it would be decentralized in the multiple databases offered by the companies that sell this storage service. It’s still physical, but yes: it works faster and with the technology that enabled bandwidth, 4g and more internet speed, we can rely on it to get and send these files from machine to database, having this seamless virtual feel.
Devices get smaller, more minimalist, and the processing and storage power can happen elsewhere. In the “cloud”. The companies and systems, are leveraged by reproducing this idea.
In the video below, you can check a recording of me interacting with my Google Home.
As a designer and technologist, my google Home’s responses to the questions “Who are you?” and “Who made you?”, bothered me a lot. They are simply not true.
The myth of the cloud does a service on behalf of those that want to keep our world unequal, to make people unaware of environmental impacts and global warming, to keep profiting with our data and to keep reinforcing biases and maintaining the same structures in power in our society. It is a weapon against critical thinking and misinformation.
And, as Google didn’t do in this case, we should not build systems intended to trick and surprise people but that they are actually involved in every step in educating them.
There are indeed several articles, books and documentaries that show the reality of the infrastructure that makes what the internet is today, but, on my references and research, I couldn’t find any interactive projects that had a different tone to break this idea of the internet as an Hyperobject.
Developing the Project
With my research as the backbone, and using as a base the content studied from the projects/books/articles of Benjamin Bratton, Kate Crawford and James Brindle, I was aiming to create an interactive webpage that turned this more academical information educating the user in an interactive way using little chunks of information and visuals.
I made some design decisions such as using Voice/Text to Speech, creating a loop and going for this this visual aesthetics for some specific reasons.
The idea of activating it through voice came mainly from the “Anatomy of an AI System” reference and of course, by the video that I showed above, on my interaction with my Google Home. I believe that by the act of asking something from our machine we humanize it in a different way, mainly when the question has its “philosophical” irony in it. The Speech to Text capability comes from a similar standpoint, to subvert the narrative that we are used to hear from machines – very functional, straightforward answers – and adding this irony of the machine revealing its true self.
The loop (and the idea of a loop that doesn’t show any progress bar or idea of user status) comes from the basic acknowledgement that even though I made my research it is still the very tip of the object: the structures behind the cloud are indeed multiple, and complex, and I don’t want to pretend I will be able to map every single aspect that is part of this system. Besides, the idea is to make the user feel as there is more to be discovered, not to overwhelm or actually map everything, but to arise the interest and believe that after going through 2 or 3 loops people will be more skeptical and rethink about the Myth of the Cloud.
This aesthetics look and feel came from the idea that I wanted to create something “brandless”, but that still made users rely on it somehow. I thought about going with some more gamecky or “hacker”/terminal approach, but I though it would lose some of its seriousness. I used this website as a visual reference.
This research has the aim to map what is unfindable on the internet and identify how online presence is distributed. Is internet a privilege?A resource? A need? A curse? What does the internet represents today both for the 50% of the world population that have direct access to it and mainly to the 50% who doesn’t.
Who is not online?
I explored the idea of identifying who we can’t find online and the systems that surround this situation. I talked to experts and people that had done almost anything possible to erase their names from search engines and social media, I read documents and listened to podcasts on the dark web, the right of prisoners to be online, asked around for people to google dead relatives from past generations, and, of course, read a lot of UN and Global Economic Forum reports. With that, I arrived in some specific groups that either are not online at all or have a reduced online presence when compared to the ‘average secular human’ today. These groups are:
Data Privacy Advocates;
Criminals that don’t use the mainstream web;
Dead people from past generations;
People that are in a social situation that does not allow them to be online (incarcerated prisoners, Orthodox Jews, Orthodox Mormons);
People that are in a social-economic situation and region where there is no access to the internet whatsoever.
By identifying these groups I aimed to create a less blurry image of what internet represents in the world today.
My classification is not complete and I do not have a quantitative stat of how much each specific group represents in numbers. A lot of times I reached the question of “what’s the difference between not having access to the internet and not being able to be found on the internet”. As I could not answer it, I decided to understand that the intention of not being online, even if idealized, was already somehow part of a movement that I should include in my research.
I became interested in analyzing the paradoxes I saw by these classifications. There is the obvious fact in my analysis I created a unique group that includes both religious people and incarcerated people – which is very very questionable, but they do usually live in developed regions where you could have internet access but still do not due to either a social imposed restriction or a shared value. And, most importantly, there is the fact that in the spectrum that comprehends the ideal of not being online (Data Privacy Advocates) and the reality of not being online (people living in regions with no internet access) there is such a huge gap of development, resources and education that somehow illustrates the inequality and the extremes of the world we live in.
And, as I saw the UN and World Economic Forum reports, I was stuck with the idea that more than 50% of the world doesn’t have internet access. I live in such a bubble of tech-savvy people, designers, technologists, developers that we never even stop to think about it. I mean, at least I didn’t. And, as I asked people from my school and work here in New York and even friends in Brazil, we have this generalized idea that internet has been fully democratized — for better or worse. But that’s simply not true. Owning a smartphone, having broadband and actually having simple internet access is still a privilege.
By that, I do not mean that the work and fight for Data privacy rights is not relevant today. On the other hand. We are still on the edge of discovering how much our connected devices keep track our information and can directly influence society in a scenario where companies and governments have total access to our personal data. There is a need to regulate and educate internet users towards it and find ways to regain our data privacy. In truth, these paradoxes are a result of the same dynamics of power that feeds itself and this world of extremes.
Still, for the purposes of focusing my research, I decided to explore more about the non-online world.
The Archipelago of Disconnection
By keeping in my mind that in fact more than half of the world doesn’t have access to the internet, I wanted to see through some visualizations what does that represent. Still, I couldn’t any that would give me a satisfying idea of that, as you check below.
Using the data from the ICT Indicators Database I started exploring some ways to visualize that with the Mercator projection, you can access it by clicking in the GIF below.
As you can note I inverted the default structure of how these maps show data: instead of highlighting the amount of online population I showed the amount of people that lacked internet access. Also, I think that the hovering interaction is more playful and easier to compare. Try doing a mouseover in Australia and then make a mouseover in Madagascar. I think it shows a lot of the paradoxes I talked about earlier.
Still, I felt like exploring this idea of map, by highlighting even more the countries with less than 10% of internet penetration, territories that are largely left out of global digital connectivity.
Therefore I created the interactive map below of the “Archipelago of Disconnection” (you can also click in the GIF to interact with it).
The map highlights an archipelago of land that is mostly disconnected from the internet and thus largely barred from participating in the cultural, educational, political, and economic activities that it affords. This archipelago of disconnection has its centre of gravity in the Sub-Saharan Africa where 18 countries have Internet penetration rates beneath the 10% . Among these Sub-Saharan countries with very low connectivity there are some very large populations – such as the Democratic Republic of the Congo (68 million), with an overall Internet penetration of a mere 4%.
After these experiments where I looked through the numbers on internet penetrations provided by International Organizations by mapping its data, I realized that I wanted to approach and understand this information both from the data creation perspective (maybe gathering the data flows and understanding the amount of information online generated from these territories) and add a more qualitative and sensitive feel to what I wanted to create as a result of this research.
With that, I remembered of the book Invisible Cities by Italo Calvino and wondered how could I connect this Fantastic realism of the narrative and the Invisible Online Territories from my Archipelago of Disconnection.
Therefore for the next step of my research I will be exploring the data available online on and from some of the cities inside the 18 countries with less than 10% internet penetration rates, trying to create a new way to visualize and highlight this “hidden information” using Italo Calvino’s Invisible Cities narrative style as a base – maybe even creating a re-reading of the book.
This looming and unequal wealth explosion is important because it will exacerbate the current fault lines of global inequality. Internet use is overwhelmingly concentrated in advanced economies, and the biggest gaps are in the world’s poorest areas.
The map and chart below shows hot spots of internet connectivity in most developed countries and huge opportunities to increase access to the internet in developing countries.
Yet we are still learning what “identity in a digital world” means. We are also still evolving policies and practices on how best to collect, process or use identity-related data in ways that empower individuals without infringing on their freedoms or causing them harm. There is significant room to improve how identity data is handled online, and how much control individuals have in the process.
Nothing is as fundamental to human beings as identity. Our identity is, literally, who we are: a combination of personal history, innate and learnt beliefs and behaviors, and a bundle of cultural, family, national, team, gender or other identities. However we understand it, identity always matters. Our identity is important because it exists in relation to others. It exists in relation to the economic and social structures in which we live. How we are represented in economic, political and other societal systems – and our degree of choice and control as to how we are represented in these systems – sets the parameters for the opportunities and rights available to us in our daily lives.
Whether we want it or not, our identity is increasingly digital, distributed and a decider of what products, services and information we access. This identity online is not simply a matter of a website login or online avatar – it is the sum total of the growing and evolving mass of information about us, our profiles and the history of our activities online. It relates to inferences made about us, based on this mass of information, which become new data points.
Today, the average internet user has 92 online accounts, and is likely to have over 200 by 2020
>>>>>> HOW MANY LOGINS I HAVE?
The result for individuals is a decreasing understanding of or control over how they are represented online. With that digital representation determining so much of how we live our lives, these changes add up to a rewriting of the social contract, and we are barely even aware of it. Any discussion on shaping digital identities should start and end with the individual – one who is born into a fully digital world – and what these identities mean for that person’s future.
One of the reasons we seldom discuss the issues of this invisible infrastructure is the fact that the speed of the packets traveling through the network is so big and unnoticeable to us, in most cases we don’t feel a significant difference in whether our packets are traveling just around the corner or to around the world and back.
There is three basic network structures: Centralized. All the devices are connected to one center. This center has privileged accessibility and thus represents the dominant element of the network. Decentralized. Although the center is still the point of highest accessibility, the network is structured so that sub-centers also have significant levels of accessibility. Distributed. No center has a level of accessibility that significantly differs to the others.
It took us just a few minutes of looking into the dataset to associate the real name of the person behind this browsing history. Just by sorting his Facebook traffic, i.e., the profile pages he visits, we were able to identify the real person. Since Facebook is enforcing a “real name policy” this is a neat way to link someone’s browsing history with their real name. For a more structured approach, there are numerous academic papers6and models on how to uniquely identify users according to their browsing patterns and behaviors. Exploring Facebook URLs reveals much more than someone’s identity. Based on the structure of the URL we were able to reconstruct a part of this person’s social graph.
We are creatures of habits, and we tend to create repetitions and patterns in our everyday behaviour. We tend to go to bed and wake up at similar times, to create our morning routines and create rituals of our social interactions. Since many segments of our lives are mediated by technology, those patterns are replicated and visible through the different digital footprints. When patterns are recognised, anomaly detection is born. As stated by Pasquinelli8, the two epistemic poles of pattern and anomaly are the two sides of the same coin of algorithmic governance. An unexpected anomaly can be detected only against the ground of a pattern regularity.
As most things today, it all started with a Google search.
This year it will make 10 years that my Grandpa passed away. And I guess that, as everyone that have lost a close relative and friend does, sometimes I like to imagine what would he be doing now and what would he think about the person I became. We were very very close, and I miss him. And, probably because of that, one day while daydreaming about him during class I randomly realized that I had never searched for his name online.
Would I find his name? Probably, he sent me a few e-mails so he did voluntarily put his name – or at least his username – on the internet. Still, it was 2008, and smartphones weren’t a thing yet and neither was social media. So yes as I laid my hands on the keyboard and typed it, I did find him, along with other 3 people that shared his same name: a businessman from Germany, a young boy from Italy, and, lastly one other person that I could not really identify the nationality but did have some interesting birds pictures on Flickr. I couldn’t find any pictures of my Grandpa though. As I looked over the only 2 Google pages as now I used the query: ‘ “Roberto+Pecis” porto alegre’, I only identified some documents from legal processes and two newspaper articles that were related to him. And that was it.
Even though the result was probably kind of obvious, it was somehow mind-blowing: his limited digital presence was such a clear representation of how much information access and production has changed in the past 10 years.
And so I started questioning myself: who can’t we find online today on the internet? what does that represent? Is it good or bad not to be online? What is the difference of voluntarily joining a social media website and having stuff online about you? What would be an actual visual representation of how much data collection increased in the past 10-20 years? Could I afford not to be online as a secular ocidental living under a capitalist system? What about people that have very common names? How hard is it to find them? What about next generations and when a lot of people starts having the same name… will we have to create maybe a global ID number to search for people online? I remember looking for friends telephones in phone books when i was a kid… but how did we actually use to find people before the internet? Does internet accessibility relates to internet ‘findability’ of people and representation?
During my Hello Computer class, I developed a bunch of experiments such as the emoji story creator, invisible map and the dada poem reciter. Those were more fun and simple weekly assignments projects, where I learned a lot and helped me explore the use of voice interfaces.
Therefore, for my final for this class I decided to create a bot using dialogflow in order to create a functional voice interface tool to help people complete a specific task. My idea was to user test and also think about the UX behind voice interfaces as well as develop a platform that could eventually be used in parallel (or even more used, who knows!?) than the webpage designed to fulfill the same functionality.
So what do I – really – need help with?
My main inspiration for this project was the current Google Assistant and that amazing – but also scary – video released in this years Google I/O where the digital assistant calls a hairdresser to book a hair saloon appointment for you.
As most impressive as that is, still, for my daily tasks the best use that i make of my Google Home/Assistant is mostly to ask for the temperature, to play a song, and eventually to help decide what should I wear according to the weather. Which is nice, but I would imagine that this tool should have way more potential to be functional in my life. So what is something that I make often and would like to get done by voice command in at least a weekly basis?
Differently from the Persona in the Google’s I/O example, that wants have her haircut done and need to book hours for it, as a student something that I schedule often is Office Hours with Residents and Professors. And ITP has a website for that.
The website collects all links from the residents/professors calendars and display their Bookable Events, which you can click and book as Office Hours. Despite some professors that do a double check with an internal NYU account, as long as you have any google @gmail account you are enabled to do that.
The website does its job, but it can be annoying as you have to go back to check if the person with the skill you want is available for the day you are also available. If the person is not, you have to go to the home and to the steps again to find another resident or teacher that could help you… and so on.
So, what if I could create a voice assistant that could book ITP office hours for you?
Would I succeed on developing it? If I did, would it indeed be useful? Would people be willing to switch from using only the website and find a voice interface actually more helpful?
Setting up DialogFlow and Firebase for a first prototype
I followed Nicole’s instructions in class and set up a simple working prototype from the DialogFlow/Firebase side for creating the ITP Assistant.
My idea for now was to make the simplest UX possible that could fulfill the task of booking office hours, even if that woudn’t be good enough for the final interface, but could work as a proof of concept and then grow from there. This is the interface I aimed to create:
So that part was working, as you can check in the video below.
My code at this point was very simple.
I added a JSON file with the names of the residents (office hours with teachers too would have to wait a bit!), with their skills, and, once i had received the string inside my skills Intent, I would parse it and try to match it with the existing skills on the JSON. If it found a matching skill I would push the Resident’s name into an array, than then would be randomized, and one of the matching resident’s selected. Once selected, while the DialogFlow would be saying to ‘hold for a sec while I book office hours”, the code would run a function to get that Resident’s URL and actually book the office hours. – that then would be said back by the Dialogflow with the date and time scheduled. You can check the code here.
So that’s when my problems begun…
Finding out that Google’s Calendsr API couldn’t work for me
(I’ll finish writing by tomorrow Oct 26th)
Working with puppeteer (fun!) and Booking office hours through my server
(I’ll finish writing by tomorrow Oct 26th)
Trying to make firebase and pupeteer work together (not successful)
As I hope it was noticed, I went for an ironic approach, imitating the look and feel of a Fashion Magazine. I decided to inform about this purely Marketing superfood term that emerged from the insight of using foods as a trend, spotting a light on how superficial that can be rather than an actual nutritional need.
My goal is to attract people that are the heavy users of those foods, who from my research are also highly interested in fashion and social media, and help them realize how superfoods are labeled and the problems around it, in order to rethink their behavior as consumers.
The feedback was very positive. Users at first were attracted to it because of the beautiful imagery and editorial design and as they quickly scraped through the copy and saw the headlines, would immediately understand the irony and comment it was a smart and funny approach. Therefore I am happy with the result.
This was the page users liked the most.
Besides being happy with the result I am also happy about the way I handled the process. The research part took me a while. It was important to read articles on the subject, but i the most important part was my ‘field trip’ to Whole Foods, where, after reading all articles and papers, I could analyze how the food was marketed to the public, interact with Supermarket employees and informally interview consumers themselves while purchasing the superfood products. What took me a while though was to have the insight of the idea and nail down the voice that I wanted to use. Once I figured that out, designing it was pretty fast.
I understand that I moved the idea to work in a comfort zone of mine, which is branding and design, and maybe in next project I should try to avoid that in order to experiment and learn using different languages. Still, considering we were required to develop a field guide for this issue and to have a printed version of that, it made sense to use that skill as a tool to conceive my message.
So what would be the next steps?
One of the feedbacks I had was to work better with the copy to push it to the next level. I totally agree with that, keeping in mind that english is not my first language, if I would look forward to publishing that this would definitely be a requirement.
Also, a lot of classmates suggested to “shop drop” this field guide in the Whole Foods magazine shelf near the cashiers. I would love to see people’s reactions! – It is a bit expensive to print that though, but if there are any interested sponsors out there just shout out and I’ll be happy to print and make the Superfoods Field guide available all around New York!
For the Open Source Studio Class, we were assigned to make the accessibility audit of a website. Since Itay and I are currently working in UX and QA for the next Adjacent issue (Adjacent is the online journal of emerging media published by ITP) and making it accessible is mandatory for us, I decided to audit it.
It is important to note that the page is not officially published yet, as we do have some design features to review, images to add and links to set. As we will be doing do in the next few days, this review could guide us on what’s missing regarding accessibility so we can launch it next Monday when it’s ready – and accessible!
The WAVE tool analyzes the writing code and the design issues related to the accessibility guidelines defined by the W3C in the WCAG. We can see we have several errors. I noted the absence of a header, which should be fixed. The other errors though are mainly regarding the use of the same links that is being used now as a placeholder for when all articles and pages are published.
Regarding colors, everything seems to be ok and accessible!
The website was also audited by using Mac VoiceOver. This was proven to be very hard, since there is no final content on in, and the repeated placeholders made everything confusing. I will definitely need to get back and redo this accessibility audit once we have the final version of the website.