top of page
clip1.gif

The last dance in the world​

GMDP-646-01.23SP- Creative Technology
Project#3: Creative technology as a personal practice
Project Instructor: 
Maxim Safioulline

In this project, by creating a 3D interactive website, I intend to change the reading experience of viewers when they visit my portfolio website, and the focus of this practice is on cursor movement. To implement it, I first created a 3D scene in Blender and then exported the model to import into three.js to set up the cursor interaction further.

CREATIVE TECH: Bio
Screen Shot 2023-03-20 at 11.02.34 PM.png

Create the farm scene using basic geometry in blender, including fence, lights, rocks and a axe.

1.png

Besides attaching colors to each object separately, I added two warm emissive lights on the lamps to create a cozy mood for this farm.

baked.jpg

UV unwrap each one of the objects then bake the render into textures.

1.gif

Export it and import the model in Three.js then apply the texture on the mesh.

CREATIVE TECH: Projects
2.gif

After loading the model and textures, created snowflakes (particle effects) by using BufferGeometry.

Then loaded the animated sheep model.

3DDFD216-7644-40EF-80F8-848CE943B393-filmage_AdobeExpress.gif

The results show that one of the fences is missing its texture, which should have been missed during the uv unwrapping process.

CREATIVE TECH: Projects

final output

CREATIVE TECH: Bio

Projection Mapping

GMDP-646-01.23SP- Creative Technology
Group Project: Xiaomeng Wang & Yanyi Wang
Project Instructor: 
Maxim Safioulline

generative_art_MP4_AdobeExpress.gif

In this project, we collected a set of everyday items made of different materials, such as glass, wood, rubber, and plastic, to build our 3D space.


By projecting some visual content, like animations or generative art, on those source materials using Madmapper, the mixed-reality spatial experience they presented proved that the daily items we couldn't be more familiar with also have a chance to produce digital art installation. It’s an attempt to strengthen the intimate relationship between art and life.

CREATIVE TECH: Bio
ezgif.com-video-to-gif.gif

**

The materials prepared for use include balloons, fish tanks, boards, curtains, etc.

IMG_2764.heic

**

Adjust the projection range based on the size of balloon's outline.

cloud_AdobeExpress.gif

Test#1: Daydream

After adding a series of animations to each balloon, we arranged them like a cloud. We intend to create a scene of someone standing under this cloud and visualize her daydreaming to the audience.

fish_tank_MP4_AdobeExpress.gif

Test#2: Fish tank

We are also curious how the projection would change on transparent materials. The experimental results show that the strong scattering of light by the glass material in a dark environment would affect the clarity of the projection effect.

CREATIVE TECH: Projects
IMG_3003.heic

Test#3: Another exhibition experience

To create an interactive spatial experience, we embed the data from the Arduino's serial monitor into the Touch Designer. Then the distance change detected by the sensor will trigger the image change.

WechatIMG6417.png

**

​​Setup in Arduino

ezgif.com-video-to-gif copy.gif

**

Particle effects in Touch Designer

generative_art_MP4_AdobeExpress.gif

final output

Once the distance sensor detects that the participant is getting closer to the painting projected on the wall, the painting will be diffused and magnified with a particle effect. 

We can imagine this scenario taking place in a real gallery. This form of visiting the exhibition helps visitors to learn more details about the work and, meanwhile, reduces the possibility of damaging it.

CREATIVE TECH: Projects
lucky2_AdobeExpress.gif

Test#4: boards

We also tried to project animations on these boards to piece together the word "lucky."

1_MP4_AdobeExpress.gif

Test#5: curtain

When we vertically flip over the shutters at 90 degrees, the image disappears from people's view.

4-filmage_AdobeExpress.gif

**

Especially the vertical typography performed a better effect.

2_MOV_AdobeExpress.gif

**

**

CREATIVE TECH: Projects

"A puppy who is willing to read a Shakespeare's poem for you."

GMDP-646-01.23SP- Creative Technology
Project#1: The WWW as a Rube Goldberg machine
Project Instructor: 
Maxim Safioulline

ezgif.com-video-to-gif.gif

By fetching data from the APIs, this project creates a chain reaction triggered by a button. Once you initiate it, you will receive a Shakespearean sonnet sent from a puppy.

CREATIVE TECH: Bio
Screen Shot 2023-04-20 at 6.34.50 PM.png

Flow Chart

trigger.png

Create a html page and add a button as the events trigger.

Screen Shot 2023-04-20 at 7.40.45 PM.png

Use "Dog API" as the result of the first triggered event.

fetch1.png

fetch the data (puppy images) from its URL

CREATIVE TECH: Projects
Screen Shot 2023-04-20 at 7.41.01 PM.png

A call to the second API: request a random sonnet from the “PoetrtDB”

Screen Shot 2023-04-20 at 7.40.24 PM.png

 a couple of examples of this URL in JSON format

fetch2.png

Fetch title and lines from the API with the correct format

ezgif.com-video-to-gif.gif

the final output of this chain reaction

CREATIVE TECH: Projects
ezgif.com-video-to-gif.gif

Chat with Harry Potter

GMDP-646-01.23SP- Creative Technology
Project#2: Chatbots with Simple AI
Project Instructor: 
Maxim Safioulline

In this project, with demonstrations from the tutorial video, I created a discord chatbot that talks like Harry Potter by fetching script data from the Harry Potter API. When people interact with it, they will not experience that stiff feeling of talking to a robot but very organic, even unexpectedly sometimes, as all of the chatbot's responses are based on the conversation context. People can trigger their conversations by simply saying "hi," as normally as if they are talking to a friend they are familiar with.

CREATIVE TECH: Bio
harry potter.png

Gather Data

Search on Kaggle to find out if there is a pre-made dialogue dataset on Harry Potter. The results showed that it included the names of the characters and the lines spoken, which was enough for this chatbot since those two elements were the only things I needed to build it.

Screen Shot 2023-02-23 at 8.40.26 PM.png

Train the model

Install and import the transformer modules in Google colab

harry potter data.png

Fetch the data from kaggle

Create a new API token in kaggle as to get the dataset from it.

sample data.png

Sample Data

After downloading the dataset, check what the sample data provides.

CREATIVE TECH: Projects
context.png

Context data frame

Create the context data frame, including the current speaking lines and the lines preceding lines, to generate responses based on the conversation context to establish a natural conversation atmosphere.

build model.png

Build the model

Build the model from pretraind "Microsoft/ DialogGPT-small" to avoid long training time.

run the main function.png

Run the main function and train it

**

huggingface token.png

Push model to Huggingface

**

CREATIVE TECH: Projects
model repo.png

Create the model repository

**

push.png

Push the model

**

conversationl model.png

create a conversational model card

To create a conversational chatbot, I edited the model card with the tag of “conversational”.

conversational model#2.png

Test it

**

CREATIVE TECH: Projects
create a bot in discord.png

Build a Discord bot

**

create a new Python repl.png

create a new Python repl

**

environment variables are all set.png

**

Store the API tokens for Huggingface and discord as the environment variables.

Filmage 2023-04-21_041911.gif

**

**

CREATIVE TECH: Projects
invite the bot to server.png

Connect the bot to Discord

**

1.gif

run the repl and log in the bot

**

CREATIVE TECH: Projects

 Shiba Inu In Wonderland

Course:  Creative Technology 2 - 22SP 
Project #2_ digital dream space

This wonderland belongs to a Shiba Inu who owns three quaint cottages. He lives with Ponyo, a young goldfish princess escaping from the ocean. Visit this wonderland. Many of his unexpected secrets are waiting for you to discover.

profile.gif
CREATIVE TECH: Image
wholescene#1.gif

Overlooking the entire wonderland, you can see the three main parts that compose it, including an island with tents, an open treehouse, and a quaint cottage on the side.

welcome scene.gif

To make Wonderland look more appealing,  two welcome video are played at the entrance.

cabin3.png

On the island area, you will see a campfire lits next to the tents.

cabin2.png

A shabby cottage is located on the opposite side of the island.

CREATIVE TECH: Projects
main cabin.gif

As you enter the treehouse, the central part of Wonderland, you have a chance to meet the owner and learn more about his private space.

INNER.gif

Interior of the tree house

iteration before.png

Iteration #1:

To ensure each device is functional and each interaction is reasonable, I embedded a video on the TV.

TV section.gif

After the iteration

iteration2 before.gif

Iteration#2:

Since the surreal animation helps convey the atmosphere of Wonderland, I replaced the still images with gifs as the welcome billboard.

iteration2.png

After the iteration

CREATIVE TECH: Projects

A Phobbling's Data Portrait

Course:  Creative Technology 2- 22SP 
Project #3_ Mediated Extension of Self

Everything about social media is designed to grab the user's attention. When you subconsciously trigger an app, you are always drawn to its design, just like some of the addictive deployments in Las Vegas that keep you in their casinos longer. People can't help but be immersed in social media, even when driving, having dinner with their family, or chatting with others. This symptom of staring at your phone screen is what we call "phubbing".


In this project, we explored how digital technology can expand and enhance our senses, bodies, and perception of ourselves. By creating a device that collects data about the time I spend on my phone, I collected and distilled seven days of phone usage data, ultimately presenting them in a visual form. It's what I called Phobbling's Data Portrait.

1.gif
CREATIVE TECH: Image
Screen Shot 2022-03-12 at 9.28.53 PM.png

#1 Coding in Arduino

In this project, I chose to use a touch sensor to detect if the phone was taken away. Specifically, if the phone is detected to be on the device, it means I am concentrating on my work. Otherwise, I might be engrossed in social networking.

WechatIMG4015.jpeg

#2 Hardware Assembly

WechatIMG4018.jpeg

#3 Connect with Adafruit 

 Test

Start Data Collection

ezgif.com-gif-maker (1).gif

When the value equals zero, it means that the cellphone is not on the device. The reverse means the cellphone is placed on it.

Screen Shot 2022-04-20 at 8.37.31 PM.png

#4  Import the collected data into visual studio.

Screen Shot 2022-04-20 at 8.40.20 PM.png

#5. Create the pattern to be displayed on the interface.

CREATIVE TECH: Projects
3.gif

#6. I use mouseEvent to randomly change the background color by moving the mouse position to enhance the visual effect.

4.gif

output

2.gif

#7 
By moving the slider left and right, you can adjust the amount of data presented in the interface. And the size of the data quantity will change the appearance of the "portrait."

CREATIVE TECH: Projects

Apple Killer

Course:  Creative Technology 2 - 22SP
Project #4_ Building For Play

Apple Killer is inspired primarily by the classical puzzle video game Angry Birds. Different from it, in this game, the slingshot is launched from the bottom up of the screen, with the intent of smashing the apples thrown in the air. To make the comical style look more straightforward to understand, several main elements of the game, slingshot, stones, and apples, are from my lame hand drawings

output.gif
CREATIVE TECH: Image
arduino.gif
hardware.gif

#1 Coding in Arduino
In this project, players could slide the joystick left or right to control the launch direction. Press the joystick vertically to fire the bullet.

test.gif
processing.gif

#2 hardwares assembly

  • Arduino Uno

  • Joystick

  • Breadboard

#3 Fix the joystick on the handle

output.gif
tester.gif

#4 

  • Coding in processing 

  • Import the background image and three main elements into Processing

  • Establish and process serial communications

  • connect with Arduino 

  • Adjust the control mode of slingshot from "keyboard control" to "joystick control."

Interface

playtester.gif

Game Testing

Game Testing

CREATIVE TECH: Projects

Defend the cookie

Course:  Creative Technology 1 - 21FA Sec 01
Project #3_ machine learning

In this game, I intended to control the puppy's movement through the machine learning Speech Commands model. Players can help the puppy jump over the oncoming cookies by issuing the command "up"!

cookie.gif
CREATIVE TECH: Image

After creating a new Javascript file "puppy.js", I forgot to reference it in the HTML, which caused the "undefined" problem.

c1.png

To make this game more enticing and engaging, I added a background and changed the rectangle to an image controlled by the player.

WechatIMG3480.jpeg

In order to give the "puppy" enough time to react to incoming cookies and then skip over them, I reduced the probability of adding cookies. That is to say, the frequency value was changed from 0.03 to 0.006. This subtle change greatly improved playability and improved the game experience.

WechatIMG3481.jpeg

Initially, I intended to use the PoseNet to control the puppy's movements by waving hands. Whereas, when I realized that my movements could not be captured flexibly by the camera, there was a delay in the puppy's movement, which would lead to a poor gaming experience. I decided to use the speech command model, the backup plan, to manipulate the game proceeding smoothly.

WechatIMG3485.jpeg
CREATIVE TECH: Projects

Caterpillar

Course:  Creative Technology 1 - 21FA Sec 01
Project #3_ interactive system

This game is based on a classic mobile game--- Snake. There are not many much-sophisticated skills that need to be maste. All the caterpillar can do is move up, down, left, and right. Eating, growing, endless repeating is the crucial mission of it. Caterpillar grows longer each time it eats.  However, unlike the traditional snake game, it would be over once the Snake hit the wall or bite its tail. The caterpillar is only set to die by hitting the wall, which reduces the difficulty of the game.

Caterpillar
CREATIVE TECH: Image

 DIAGRAM:

The running system of caterpillar

I use different shapes to clearly distinguish between caterpillars and food.

To make the game easier to play, I adjusted the parameters of framerate so as to reduce the speed of the caterpillar's movement.

Untitled.jpg
WechatIMG3486.png

From investigating the use of the "Array.shift" function, it's feasible to help me present changes in the growth of the caterpillar's body.

WechatIMG3487.png
WechatIMG3488.png
CREATIVE TECH: Projects

Bubble Bath

Course:  Creative Technology 1 - 21FA Sec 01
Project #1_non-linear movement

Countless bubbles will appear in front of players eyes. They will take up the entire screen, which may induce an aversion to people who have Trypophobia unless they keep clicking the mouse.

bubble bath.gif
CREATIVE TECH: Bio

To create a screen dimension for this project, I used the "windowWidth" syntax.

To create the colorful bubble effect, I used the random() syntax to randomize the color and size of the bubbles.

As the transparency of the background color is adjusted, the background color begins to overlay as the code loops, resulting in a gradient effect.

WechatIMG3492.png
WechatIMG3489.png

To enhance the interaction between the audience and the project, I use the mousePressed() function, which allows the them to control the number of bubbles and the color of the background by clicking the mouse.

WechatIMG3491.png
WechatIMG3490.jpeg
CREATIVE TECH: Projects

Bézier curves

Course:  Creative Technology 1 - 21FA Sec 01
Project #1_ organic shape

In this project, I used the Bezier function to create a series of smooth and rhythmic curves.

Perlin noise.gif
1.1.png

When I set the incremental value t=0.1, the motion is not coherent and you can even see each Bezier curve. Then I changed this value to 0.004 and the motion became smoother.

 I chose to use the "slider feature" to provide some interactive behavior for the user. The background color will be changed by moving the slider.

2.1.png
CREATIVE TECH: Projects

From using the “frameCount” function, the background will be cleared in every 200 frames.

B4.png
last.png

By changing the transparency value of the stroke, the edges of the Bezier curve are blurred on the canvas.

CREATIVE TECH: Projects

Twinkle, Twinkle, Shine

Course:  Creative Technology 1 - 21FA Sec 01
Project #5_single-channel controller

CREATIVE TECH: Video
sketch.png

#1 Sketch

Draw a sun in "Processing" to create an interactive interface.

WechatIMG3442.png

#2 Coding in Arduino

For this project, I chose to use a photosensitive sensor to control the light intensity of the "sun".

monitor.gif

#3 Pretest

By detecting the serial monitor to ensure that the code and light sensor are both working properly.

CREATIVE TECH: Projects
Laptop
shortcut.gif

#4 Assembling



processing.gif

#5 Coding in Processing

The code in the left section demonstrates how to get the value of the photosensitive sensor from Arduino and use the value to control the brightness of the sun. In a dark environment, the sun  turns dark, adversely, in a light environment, the sun turns bright, which presents like a "twinkle sunshine".

twinkle sunshine.gif

#6 Final Output

Interface display in Processing

CREATIVE TECH: Projects

Music Gadget

CREATIVE TECH: Video

Course :  Creative Technology 1 - 21FA Sec 01

Project 5_musical instrument

By hooking together Arduino and Processing, this project created an analog piano keyboard, establishing a physical interface to manipulate the music saved in the computer. The six buttons are used to control their corresponding six notes to achieve a simple musical solo.

#1 Coding in Arduino

Code the corresponding note for each button to manipulate the sequence in which the notes are played.

arduino.png

#2 Assembling

First, assemble six buttons, resistors and jumper wires on bread board and connect them to the ESP32. Then, cut the foam board into rectangles in an appropriate size and draw a series of  black and white keys on it.  After that, insert six buttons at the top of the white keys.

P5.jpeg

#3 Coding in Processing

  • ​Store audio files of notes into processing.

  • Establish and process serial communications     

  • ​connect arduino with processing​​

  • Sketch 6 circles to represent the notes in the interface.

  • draw a function that would be triggered every time the special character 

WechatIMG3452.jpeg

#4 test 

Check if the "system" is running properly

PRE.gif

#5 Interface