The last dance in the world
GMDP-646-01.23SP- Creative Technology
Project#3: Creative technology as a personal practice
Project Instructor:
Maxim Safioulline
In this project, by creating a 3D interactive website, I intend to change the reading experience of viewers when they visit my portfolio website, and the focus of this practice is on cursor movement. To implement it, I first created a 3D scene in Blender and then exported the model to import into three.js to set up the cursor interaction further.
Create the farm scene using basic geometry in blender, including fence, lights, rocks and a axe.
Besides attaching colors to each object separately, I added two warm emissive lights on the lamps to create a cozy mood for this farm.
UV unwrap each one of the objects then bake the render into textures.
Export it and import the model in Three.js then apply the texture on the mesh.
After loading the model and textures, created snowflakes (particle effects) by using BufferGeometry.
Then loaded the animated sheep model.
The results show that one of the fences is missing its texture, which should have been missed during the uv unwrapping process.
final output
Projection Mapping
GMDP-646-01.23SP- Creative Technology
Group Project: Xiaomeng Wang & Yanyi Wang
Project Instructor:
Maxim Safioulline
In this project, we collected a set of everyday items made of different materials, such as glass, wood, rubber, and plastic, to build our 3D space.
By projecting some visual content, like animations or generative art, on those source materials using Madmapper, the mixed-reality spatial experience they presented proved that the daily items we couldn't be more familiar with also have a chance to produce digital art installation. It’s an attempt to strengthen the intimate relationship between art and life.
**
The materials prepared for use include balloons, fish tanks, boards, curtains, etc.
**
Adjust the projection range based on the size of balloon's outline.
Test#1: Daydream
After adding a series of animations to each balloon, we arranged them like a cloud. We intend to create a scene of someone standing under this cloud and visualize her daydreaming to the audience.
Test#2: Fish tank
We are also curious how the projection would change on transparent materials. The experimental results show that the strong scattering of light by the glass material in a dark environment would affect the clarity of the projection effect.
Test#3: Another exhibition experience
To create an interactive spatial experience, we embed the data from the Arduino's serial monitor into the Touch Designer. Then the distance change detected by the sensor will trigger the image change.
**
Setup in Arduino
**
Particle effects in Touch Designer
final output
Once the distance sensor detects that the participant is getting closer to the painting projected on the wall, the painting will be diffused and magnified with a particle effect.
We can imagine this scenario taking place in a real gallery. This form of visiting the exhibition helps visitors to learn more details about the work and, meanwhile, reduces the possibility of damaging it.
Test#4: boards
We also tried to project animations on these boards to piece together the word "lucky."
Test#5: curtain
When we vertically flip over the shutters at 90 degrees, the image disappears from people's view.
**
Especially the vertical typography performed a better effect.
**
**
"A puppy who is willing to read a Shakespeare's poem for you."
GMDP-646-01.23SP- Creative Technology
Project#1: The WWW as a Rube Goldberg machine
Project Instructor:
Maxim Safioulline
By fetching data from the APIs, this project creates a chain reaction triggered by a button. Once you initiate it, you will receive a Shakespearean sonnet sent from a puppy.
Flow Chart
Create a html page and add a button as the events trigger.
Use "Dog API" as the result of the first triggered event.
fetch the data (puppy images) from its URL
A call to the second API: request a random sonnet from the “PoetrtDB”
a couple of examples of this URL in JSON format
Fetch title and lines from the API with the correct format
the final output of this chain reaction
Chat with Harry Potter
GMDP-646-01.23SP- Creative Technology
Project#2: Chatbots with Simple AI
Project Instructor:
Maxim Safioulline
In this project, with demonstrations from the tutorial video, I created a discord chatbot that talks like Harry Potter by fetching script data from the Harry Potter API. When people interact with it, they will not experience that stiff feeling of talking to a robot but very organic, even unexpectedly sometimes, as all of the chatbot's responses are based on the conversation context. People can trigger their conversations by simply saying "hi," as normally as if they are talking to a friend they are familiar with.
Gather Data
Search on Kaggle to find out if there is a pre-made dialogue dataset on Harry Potter. The results showed that it included the names of the characters and the lines spoken, which was enough for this chatbot since those two elements were the only things I needed to build it.
Train the model
Install and import the transformer modules in Google colab
Fetch the data from kaggle
Create a new API token in kaggle as to get the dataset from it.
Sample Data
After downloading the dataset, check what the sample data provides.
Context data frame
Create the context data frame, including the current speaking lines and the lines preceding lines, to generate responses based on the conversation context to establish a natural conversation atmosphere.
Build the model
Build the model from pretraind "Microsoft/ DialogGPT-small" to avoid long training time.
Run the main function and train it
**
Push model to Huggingface
**
Create the model repository
**
Push the model
**
create a conversational model card
To create a conversational chatbot, I edited the model card with the tag of “conversational”.
Test it
**
Build a Discord bot
**
create a new Python repl
**
**
Store the API tokens for Huggingface and discord as the environment variables.
**
**
Connect the bot to Discord
**
run the repl and log in the bot
**
Shiba Inu In Wonderland
Course: Creative Technology 2 - 22SP
Project #2_ digital dream space
This wonderland belongs to a Shiba Inu who owns three quaint cottages. He lives with Ponyo, a young goldfish princess escaping from the ocean. Visit this wonderland. Many of his unexpected secrets are waiting for you to discover.
Overlooking the entire wonderland, you can see the three main parts that compose it, including an island with tents, an open treehouse, and a quaint cottage on the side.
To make Wonderland look more appealing, two welcome video are played at the entrance.
On the island area, you will see a campfire lits next to the tents.
A shabby cottage is located on the opposite side of the island.
As you enter the treehouse, the central part of Wonderland, you have a chance to meet the owner and learn more about his private space.
Interior of the tree house
Iteration #1:
To ensure each device is functional and each interaction is reasonable, I embedded a video on the TV.
After the iteration
Iteration#2:
Since the surreal animation helps convey the atmosphere of Wonderland, I replaced the still images with gifs as the welcome billboard.
After the iteration
A Phobbling's Data Portrait
Course: Creative Technology 2- 22SP
Project #3_ Mediated Extension of Self
Everything about social media is designed to grab the user's attention. When you subconsciously trigger an app, you are always drawn to its design, just like some of the addictive deployments in Las Vegas that keep you in their casinos longer. People can't help but be immersed in social media, even when driving, having dinner with their family, or chatting with others. This symptom of staring at your phone screen is what we call "phubbing".
In this project, we explored how digital technology can expand and enhance our senses, bodies, and perception of ourselves. By creating a device that collects data about the time I spend on my phone, I collected and distilled seven days of phone usage data, ultimately presenting them in a visual form. It's what I called Phobbling's Data Portrait.
#1 Coding in Arduino
In this project, I chose to use a touch sensor to detect if the phone was taken away. Specifically, if the phone is detected to be on the device, it means I am concentrating on my work. Otherwise, I might be engrossed in social networking.
#2 Hardware Assembly
#3 Connect with Adafruit
Test
Start Data Collection
When the value equals zero, it means that the cellphone is not on the device. The reverse means the cellphone is placed on it.
#4 Import the collected data into visual studio.
#5. Create the pattern to be displayed on the interface.
#6. I use mouseEvent to randomly change the background color by moving the mouse position to enhance the visual effect.
output
#7
By moving the slider left and right, you can adjust the amount of data presented in the interface. And the size of the data quantity will change the appearance of the "portrait."
Apple Killer
Course: Creative Technology 2 - 22SP
Project #4_ Building For Play
Apple Killer is inspired primarily by the classical puzzle video game Angry Birds. Different from it, in this game, the slingshot is launched from the bottom up of the screen, with the intent of smashing the apples thrown in the air. To make the comical style look more straightforward to understand, several main elements of the game, slingshot, stones, and apples, are from my lame hand drawings
#1 Coding in Arduino
In this project, players could slide the joystick left or right to control the launch direction. Press the joystick vertically to fire the bullet.
#2 hardwares assembly
Arduino Uno
Joystick
Breadboard
#3 Fix the joystick on the handle
#4
Coding in processing
Import the background image and three main elements into Processing
Establish and process serial communications
connect with Arduino
Adjust the control mode of slingshot from "keyboard control" to "joystick control."
Interface
Game Testing
Game Testing
After creating a new Javascript file "puppy.js", I forgot to reference it in the HTML, which caused the "undefined" problem.
To make this game more enticing and engaging, I added a background and changed the rectangle to an image controlled by the player.
In order to give the "puppy" enough time to react to incoming cookies and then skip over them, I reduced the probability of adding cookies. That is to say, the frequency value was changed from 0.03 to 0.006. This subtle change greatly improved playability and improved the game experience.
Initially, I intended to use the PoseNet to control the puppy's movements by waving hands. Whereas, when I realized that my movements could not be captured flexibly by the camera, there was a delay in the puppy's movement, which would lead to a poor gaming experience. I decided to use the speech command model, the backup plan, to manipulate the game proceeding smoothly.
Caterpillar
Course: Creative Technology 1 - 21FA Sec 01
Project #3_ interactive system
This game is based on a classic mobile game--- Snake. There are not many much-sophisticated skills that need to be maste. All the caterpillar can do is move up, down, left, and right. Eating, growing, endless repeating is the crucial mission of it. Caterpillar grows longer each time it eats. However, unlike the traditional snake game, it would be over once the Snake hit the wall or bite its tail. The caterpillar is only set to die by hitting the wall， which reduces the difficulty of the game.
DIAGRAM:
The running system of caterpillar
I use different shapes to clearly distinguish between caterpillars and food.
To make the game easier to play, I adjusted the parameters of framerate so as to reduce the speed of the caterpillar's movement.
From investigating the use of the "Array.shift" function, it's feasible to help me present changes in the growth of the caterpillar's body.
To create a screen dimension for this project, I used the "windowWidth" syntax.
To create the colorful bubble effect, I used the random() syntax to randomize the color and size of the bubbles.
As the transparency of the background color is adjusted, the background color begins to overlay as the code loops, resulting in a gradient effect.
To enhance the interaction between the audience and the project, I use the mousePressed() function, which allows the them to control the number of bubbles and the color of the background by clicking the mouse.
Bézier curves
Course: Creative Technology 1 - 21FA Sec 01
Project #1_ organic shape
In this project, I used the Bezier function to create a series of smooth and rhythmic curves.
When I set the incremental value t=0.1, the motion is not coherent and you can even see each Bezier curve. Then I changed this value to 0.004 and the motion became smoother.
I chose to use the "slider feature" to provide some interactive behavior for the user. The background color will be changed by moving the slider.
From using the “frameCount” function, the background will be cleared in every 200 frames.
By changing the transparency value of the stroke, the edges of the Bezier curve are blurred on the canvas.
Twinkle, Twinkle, Shine
Course: Creative Technology 1 - 21FA Sec 01
Project #5_single-channel controller
#1 Sketch
Draw a sun in "Processing" to create an interactive interface.
#2 Coding in Arduino
For this project, I chose to use a photosensitive sensor to control the light intensity of the "sun".
#3 Pretest
By detecting the serial monitor to ensure that the code and light sensor are both working properly.
#4 Assembling
#5 Coding in Processing
The code in the left section demonstrates how to get the value of the photosensitive sensor from Arduino and use the value to control the brightness of the sun. In a dark environment, the sun turns dark, adversely, in a light environment, the sun turns bright, which presents like a "twinkle sunshine".
#6 Final Output
Interface display in Processing
Music Gadget
Course : Creative Technology 1 - 21FA Sec 01
Project 5_musical instrument
By hooking together Arduino and Processing, this project created an analog piano keyboard, establishing a physical interface to manipulate the music saved in the computer. The six buttons are used to control their corresponding six notes to achieve a simple musical solo.
#1 Coding in Arduino
Code the corresponding note for each button to manipulate the sequence in which the notes are played.
#2 Assembling
First, assemble six buttons, resistors and jumper wires on bread board and connect them to the ESP32. Then, cut the foam board into rectangles in an appropriate size and draw a series of black and white keys on it. After that, insert six buttons at the top of the white keys.
#3 Coding in Processing
Store audio files of notes into processing.
Establish and process serial communications
connect arduino with processing
Sketch 6 circles to represent the notes in the interface.
draw a function that would be triggered every time the special character
#4 test
Check if the "system" is running properly
#5 Interface
The interface presented in "Processing"
final presentation
Other Works
Rhythm
Don't eat toooo much.
Glowworm
Bouncing Ball
